Risk Based Testing: Identification, Priorities, Matrices and Approaches


What Is Risk Based Testing?

The risks are present in several areas of knowledge, but they have a distinct meaning in each context. The generalized definition of the term, according to Wikipedia:

It’s the possibility of something bad happening. It involves uncertainties/implications about the effect of an activity on something that humans value, usually focusing on the negative and undesirable consequences.

In the context of software engineering, we can relate the risks directly related to project management as a whole or strictly focus on the use of the application.

So, we can focus on the possible negative experiences that can be perceived by the user of the product. That’s where the risk-based testing approach comes in.

According to a ReQtest article, this technique checks the probability of application failures, by using previously created test cases to predict the impact of what has been developed on the customer’s business.

Besides that, critical parts of the application are prioritized, where the probability of crashes is higher and where effort should also be invested in fixing critical bugs.

Application of risk-based testing

Depending on the context (traditional or agile) there are differences in how we collect data.

The function performed and the artefacts produced. A more detailed and general approach to the application is discussed below.

Step 1: Risk Identification

To identify risks, we can perform:

Brainstorming meetings, involving, if possible, all areas participating in the project, both the technical area and the business area or even the user himself. The purpose of the ceremony is that each one identifies a risk in the application/ project.

Known crash/Risk history

How many times has the deployment of certain functionality generated bugs in production? What is the most problematic feature in recent months? Which part of the system generates the most complaints from users? By collecting this information, we may understand the basis of the application and the points that we need to observe the application more carefully and possible risks already identified previously due to the history of software problems.

Inside OutStrategy: 

Repeatedly ask the question “what can go wrong here?” Responses should take into account possible flaws that may exist in the application, vulnerability situations, and also the parties who will be most impacted by the possible failure.

Outside In Strategy

Risks are explored based on quality models, such as NBR ISO/IEC 9126, and for each topic of this model the question is asked: “What can happen if this requirement is not met?”.

Evaluation of the complexity of the application or functionality: Due to the level of complexity of what has been developed, what can go wrong? How does this functionality influence the system?

Step 2: Prioritizing risks

After identifying the risks, they should be prioritized. Commonly used approaches are the probability-impact ratio and the risk matrix.

Probability x Impact

For each risk, we give value, from 1 to 5 for probability and impact. The lower the value, the less likely the situation occurs, and the lower the impact for users/businesses.

Subsequently, the multiple probabilities and impact value are resulting in the risk factor. Which we can use for prioritization.

Higher-scoring risks need more attention and will require more testing time.

Generally, we can define the probability by the product technical team and the impact of the business area.

Risk matrix

The risk matrix follows the same line of reasoning as the previous table. However, we can use the low, medium, and high keywords instead of values to determine the prioritization of activities, as shown below:

Step 3: Define risk mitigation strategies

With the risks properly identified, we know where to invest efforts in testing. Using the previous example, we can associate each system module with specific risk, for example, risk 1 concerns module 1.

Risks 4 and 3 had the highest scores, so they should be the focus of the strategy. The mitigation plan can include:

  1. Define which testing techniques can be used to cover the risks.
  2. Create test cases specifically targeted to cover the risks encountered;
  3. Prioritize the execution of all test cases associated with these modules;
  4. Perform a more in-depth analysis of the impacts on the modification of these two modules in the production environment, taking into account the history of failures.
  5. Negotiate the increase of the deadline for the delivery of the project.
  6. Other points may also include in the mitigation plan, all depending on the context of the project.

Risk-based testing in the agile context

In an InfoQ article, Ben Linders reports Csaba Szökőcs’ experience in implementing risk-based testing in an agile team that uses Scrum.

Csaba says that the teams try to collect the risks of each story before sprint planning and that helps them to think about testing activities before they implement the stories.

Once they collect all the stories, two testers prioritize risks through the probability-impact ratio and they will treat all critical points with high exposure in the same way.

You adopt this type of test when you have a short time to test and that is done by the test teams to find the most critical cases that you need to test.

Where testers run separate, independent tests based on the risks identified for almost all user stories.

We collect the risks in the context of the user’s story by answering the question, “What can go wrong in this story?”.

Besides, we try to imagine that the new feature is already in operation. We think of different scenarios, how it will be used correctly, and how it will be misused.

We tried to combine the new feature with other existing features, answering the question, “Can it go wrong in any way?

Pros and cons of adopting risk-based testing

Henriette Harmse lists some advantages and disadvantages of adopting this technique:

Advantages

  1. You must develop and test the most critical items first, which will reduce the overall risk in the project.
  2. If the deadline is tight, there is a guideline to guide what you should test and what you should not test.
  3. At any time, we can find new risks in the project. Thus, risks can be prioritized again, taking into account the most critical items that have not yet been tested;
  4. The identified risks are valuable information for all who are involved in the project and can use in negotiations.

Disadvantages

  1. With this approach, we can’t test the parts of the system intentionally.
  2. The values attributed to the risks can be subjective, so it is necessary to discuss with the team the proposed values;
  3. You can increase the documentation load.

Risks vs. Uncertainties

The risk and uncertainty in the application of the risk-based testing approach and the entire content of this section are based on this article.

According to Frank Knight:

They are reserved for scenarios where they can be measured quantitatively, usually using probability;

Uncertainty: situations that are not quantifiable or measured, with the tools that the team currently has.

Knowing the differences, we can build a matrix that brings the relationship between these two concepts, as shown below:

James states that the risks are associated with the first quadrant (known), where the whole team is aware of the problem and understands all its nuances to the point of measuring it quantitatively.

In all other quarters, there is a level of uncertainty that can be unraveled with the testing activities and further risk analysis testing of the situation. The idea is to turn all levels of uncertainty into something known and measurable!

Risk-based testing or risk assessment helps prioritize efforts in stories and critical parts of the application, and there are several ways to put this technique into practice. Risk analytics assists take the guesswork out of controlling risk-related problems by using a range of techniques and technologies to conclude insights, calculate likely strategies, and foresee future events.

Plotting a good plan for how to adopt this strategy in your team will help you in anticipating various situations that would only be noticeable in the production environment, which causes a negative product experience for the end-user.

Already using RBT or would you like to use it? Do you think it’s worth adopting this practice? 

Tell us in the comments!