Risk Based Test Design

Okay, we know we want to do risk-based testing. What does that mean about the work we’re going to do?

Several months ago our CTO gave a talk talking about the concepts of software architecture.

The core of the talk was this:

  • Principles:

  • Models:

  • Standards:

So here’s what we want to do in our testing strategies.

Step 0: Get our hands on the system under test to start learning the context we’re working and thinking in

Step 1: Identify principle risks; these are the risks that are direct blocks to revenue.

Step 2: Brainstorm some workflow risks; things that could go wrong specific to the workflow we’re responsible for evaluating (note: every workflow we evaluate should contribute to overcoming at least one principle risk)

Step 3: Get our hands on the system to update our mental model for the problem. Are the risks we’re thinking of feasible? Do other people agree that these would be negative outcomes.

Step 4: Brainstorm action risks; things that could go wrong for the person or system responsible for executing part of the workflow.

Step 5: Prioritize those risks in terms of likelihood and impact.

Step 6: Execute experiments, continuously updating the mental model and refreshing the list of risks

There’s three fundamental levels of risk:

  • Principle
  • Workflow
  • Action

I want to work the Action Risks with my hands, see the Workflow Risks in my peripheral vision, and orient my body facing the Principle Risks.

==================

Subway Case study (100% fictional):

Principle Risks:

  • We don’t drive repeat business
  • We don’t drive new business
  • We make somebody sick
  • We waste product

We happen to know that Subway’s business analysts think the freshness and quality of the bread is key to mitigating the top two principle risks.

15 years ago Subway signed a contract with Oven Inc. Oven Inc has since gone out of business and so as franchises age out of their Oven Inc ovens, Subway’s corporate office needs to roll out the new solution.

Let’s think about workflow level risks. These are things that could go wrong with the bread-making workflow. Of course the making of the bread involves more than the active steps of putting the dough in the oven. So, we need some dual brainstorming here. A brainstorm for the workflow and a brainstorm for what could go wrong with the workflow. To my mind, the brainstorm of the workflow represents what Rapid Software Testing calls a product coverage outline (PCO). How could I possibly brainstorm risks around something I can’t describe or understand?

Workflow:

  • procurement of raw materials
  • mixing of dough
  • proving of dough
  • storage of dough
  • preheat oven
  • retrieval of stored dough
  • dough into oven
  • baking time and temperature
  • removal of bread from oven
  • treatment (?) of newly baked loaves
  • storage of newly baked loaves
  • movement of leftover loaves
  • oven off
  • disposal of any waste
  • report of inventory used

questions:

  • who owns each step? Corporate or franchise?
  • How often are quality checks run on franchises?
  • How long can dough be stored before baking bread?
  • What are the quality metrics currently in play to measure this process?

So staying at the workflow level, the workflow being “make the best bread we can,” what are some things that can go wrong:

  • Bread of a certain type gets underbaked
    • because instructions aren’t clear
    • because the oven is complicated
    • because the oven alert sounds just like some other alert, so the employee pulls out the bread according to the wrong visual/audio cue
  • Bread of a certain type gets overbaked
  • Customers want a certain type of bread that I don’t have
    • Either I’m out for the day OR
    • I have dough on hand but didn’t bake new loaves of that bread
  • We have to throw away good loaves of bread because they get stale on the shelf
  • We have to throw away dough because it expires
  • The oven is time consuming to set up, so wait times are increased (which we know from business analysts make it less likely to generate repeat customers)
  • The storage of bread is time consuming, so wait times are increased
  • The dough is no good, but the franchise employee has no way to know that (so our bread makes people sick)

Moving on to action risks:

===

So, in 30 minutes, I have some naive idea about the risks I could investigate if Subway hired me as a contractor to evaluate their “new bread” processes. I start with principle risks – how does this company/product/service make money? At the revenue generation level, what could go wrong? What general categories of “losing money” exist in this business?

Then I work on to workflow risks – what are the inputs and outputs of this workflow, and how does it contribute to the mitigation of the principle risks.

Finally, I decompose the workflow itself. For each step in the process. What could go wrong as we attack this step?