End the war with flaky tests with Gunjan Arora
Meet Gunjan Arora, our Principal Quality Engineer at Nine. She shares her tips on beating flaky tests.
Picture this – You’re just finishing your morning coffee as your automated test suite pings to indicate that it’s completed its run job. You notice that a couple of tests have failed and you check to see what they are with curiosity … and then your heart sinks as you realise that it’s a test that has gone off incorrectly in the past.
Is it really a bug?
Or a false positive?
If you have experienced this, you will know the pain of a flaky test.
A flaky test is a test that could fail or pass for the same configuration. Ideally, a test should constantly pass or fail if no code changes are applied. So a flaky test could be really annoying to engineers because the failures do not always indicate bugs in the code.
They can also be quite costly since they often require engineers to retrigger the entire jobs on a CI pipeline and often waste a lot of time waiting for them to complete successfully.
The real cost of test flakiness is a lack of confidence in your tests.
There are a few things you can do to help identify flakiness in your test suite.
- Look for timeouts. If you have a complex test relying on asynchronous services, try to check the service for availability and stability before starting the test.
- Visualise test runs. A simple analytics report allows teams to see their tests, see which ones are fast & slow, and see which ones are flaky.
- Use tools to detect flakiness. There are tools in the market or you may write a custom bot to help you identify the flaky tests.
How to deal with flaky tests
- Don’t just blindly rerun the random failures, instead quarantine them and systematically fix them.
- Understand the nature of failure. Sometimes a simple fix like adding a retry, adding a custom wait method or updating the element selector may fix the problem.
- A test that fails is not always flaky, so instead of thinking of it as a test code issue, also verify for any bugs in the application or environment causing the failure.
- Run the test in isolation multiple times to understand the common theme if the failure is due to a particular browser, device or particular functionality.
- Do not get overwhelmed with failing tests, it’s always good to pair with another engineer to get a fresh perspective.
The only thing to keep in mind is that when a test fails, there is definitely something wrong. Flakiness is not a fluke. Trust your tests and start debugging, so you can avoid the distress with your next test job run. More importantly, to enjoy your morning coffee!
Read our Content
Q&A with Maddy Callender: Senior Producer – Wide World of Sports
A unique Nine story, working on the National Rugby League, championing the Beanie for Brain Cancer round with the Mark Hughes Foundation and many other magic moments with the WWOS team.
Q&A with Gabi Richardson: Content Partnerships Executive – Sport / Powered
Gabi works with the program production teams (e.g. WWOS) and the sales teams to find commercial opportunities that compliment our sports coverage. She is responsible for implementing in-program integration within our tennis broadcast.
Q&A with Scott Spits, The Age Reporter
Scott is a sports reporter and producer with many years of experience, involving a range of tasks including liaising with other reporters, quality control and presentation of our journalism.
Guide to Interviewing at Nine
While interviews can be incredibly nerve-wracking, this is the occasion to show what your resume does not. It is...
Make Your Resume Stand Out at Nine
It's essential to make your resume stand out from the rest. Here are some key strategies to help you achieve just that.
Component Testing and Test Driven Development with Jack Donaldson
Meet Jack Donaldson, one of our Software Engineers at Nine. Jack shares some insights on testing.