Hi, I think there’s a lot of people who’ve been in the same trouble as me because the automated test we wrote is flaky. Today, I have five reasons to tell you why your test is turning out to be flaky.
But first, let’s understand what a Flaky Test is.
The word Flaky Test in Software Development means that your test is so unreliable that every time you run the compilation, it returns a different result. Sometimes it runs through and sometimes it doesn’t, depending on the presence of code solving applications, which is a problem that almost all organizations have come across.
Many people who invested in automated test writing hope that it will reduce the effort in the manual test. After you repeatedly get undesirable results, you will find that having a flaky test is more of a headache than you thought it would be. Hence, it is very important for you to recognize why this might be happening.
Here are some reasons why your test is turning out to be flaky.
1. Improper Test Level break
In good test principles, we should divide our Test Level according to the Concept Test Pyramid.
I believe that there are still many people who are still stuck in the End-To-End Test (E2E) Tests. which have many disadvantages, for example.
- The test takes a long time to run because it requires a test in the real Black Box Testing User Perspective view, and most of them are UI Testing.
- The test requires high dependency, such as having a server-side (backend) that is always available
Therefore, the proper Test Level divide will help us focus on the features we can. We should focus on unit test and reduce the amount of end-to-end test. That is the only critical path with large features that must always guarantee functionality to work properly, where the proportion of Test Level from Unit/Integration/E2E should be 70:20:10 (Reference based on Google’s recommended proportions) which, the better we do, the better. No only are the likelihood of flaky tests less likely, the tests will also run faster.
2. Third-Party Dependencies
This is a classic issue that many people should have met: our Application has a 3rd Party Application that we can’t control.
If a third-party application has changed or is in development while we want to test, such as the case of backend/frontend mentioned earlier, it will definitely cause Flaky Test problems. The only solution is that we need to bring the concept of Dependency Injection (DI) such as Mock/Stub/Spy. This lets us get the most out of our application-related dependency. For example, Network Requests, Data Storage/Caching Layer, which will have a positive effect on us to control all the factors that will cause the Flaky Test.
3. Large Test
The test writing is too large. As a result, flaky Test writing is a lot of long-running (end-to-end test) that gives Test Case one with step in a too-long Test, which is even more likely to have Step.
The solution is that we need to write a smaller test to reduce setup step.
Turn to more Dependency Injection, make a test in a spot that should be tested, not to write 1 Test for everything (Test 1 should have a clear objective of wanting to test one of only one).
4. Infrastructure Problem
Many people may have come across the problem of writing and running tests on their machines and then through them all the time. But why is it running on the Continuous Integration System and it breaks all the time?.
This problem is caused by environment configurations such as operating system, software/library version differences between Development Environment and CI Environment.
5. Test Framework
Is the Test Framework one of the things that makes the test Flaky? Choosing a Test Framework that is not appropriate for our Application is one of the things that can cause a flaky Test. For example, someone who wants to do an API Test, but writes as a Web Interface to cover another layer to make a simple API Call through pressing the button and using the Web Test Framework to make tess. It’s not ideal, it’s better to use the Test Framework API directly.
A good Test Framework should require a feature or tool to help keep our Test running stable, such as supporting dependency injection with the Test Framework itself, access to our objects, state application at run time. This is so that we can do more low-level tests, and have the ability to do a good debugging to help people write tests Investigate problems when Test Fail is easy.
Finally, the most important thing is that test writers should have to have a good understanding of the functionality and structure of the application that wants to test as well.