Code review in the pursuit of quality assurance (QA) is a hot topic among the developer community. Software engineers have an amplified (but arguably very justifiable) proclivity for discussing the systematic procedures and methodologies they adopt to ensure live production software is maintained correctly, and to guarantee newly deployed code falls harmoniously in line with the working codebase. All of which efforts are designed to keep code bugs at bay. Except things don’t always work out that way.
Ask any given software team how their software quality assurance ratings are, and they will usually point to all the cumbersome bugs that they have caught.
The dashboard is green, the release went out, the incident channel is quiet. What that picture leaves out is the failures that never raised their hand.
Loud vs. quiet software bugs
QA engineer at a high-load e-commerce platform company Mikhail Golikov says that, in his experience as a tester on systems with more services than people, those quiet failures are where quality is actually lost.
He defines a difference between “loud” software bugs i.e. ones that look more tangentially erroneous and incongruent and therefore are more commonly caught vs. “quiet” software bugs that never make a sound i.e. tests that exist but never run, flaky tests everyone has learned to ignore and behaviour that lives in production logs but in no test at all.
“A loud failure is the easy case. A test goes red, a build breaks, someone gets paged. The system is doing its job: it found a problem and refused to stay silent about it. You can argue about priority, but you cannot ignore what you can see. The failures that hurt are the ones that pass right through that net because nothing was ever wired up to shout,” explained Golikov.
The truth of the matter is, it’s the same three problems that seem to come up time and time again:
- The test that exists, but never runs.
- The flaky test everyone has agreed to ignore.
- The behaviour that lives only in the logs.
Looking at the reality of the test that exists but never runs, Golikov says that most teams have more tests than they think and fewer than they need, and the gap is full of checks that simply never execute.
When documentation wears a test’s clothes.
“The classic version is a drawer of API collections, captured carefully in a tool like Postman, that only run when a human opens them and clicks. The requests are real, the expected responses are real, the auth is figured out. The one thing missing is the part that makes a test a test: it has to run without you, and fail loudly when something breaks,” said Golikov. “A check that only runs when someone remembers is not protecting anything – it is documentation wearing a test’s clothes.
He recounts an example where he ended up writing a small tool to turn those collections into committable pytest files (Python scripts containing automated test cases designed to systematically identify software bugs and verify code correctness) precisely because the work was already done; it just was not being run.
Flaky flaky, leads to achy breaky
Logically then, we come to the flaky test that everyone has agreed to ignore – and this one is worse, because the team has not just missed the failure; it has trained itself to look away.
“A test that flips between pass and fail without any code change is a test the team stops believing. First you re-run it. Then you re-run it by reflex. Then a real regression hides inside the noise and ships, because the red was assumed to be the usual flakiness. The quality cost is not the flaky test itself; it is that flakiness quietly lowers the trust threshold for every red build around it. You cannot fix what you have decided in advance to disbelieve,” clarified Golikov.
Thirdly, then, the behaviour that lives only in the logs.
This is a scenario that exists where every running service narrates what it actually does, request by request, in its logs. Some of that behaviour is covered by tests. A lot of it is not, and the gap between what a service does in production and what the software suite checks is invisible until something in that gap breaks.
“The logs are sitting right there, a record of real traffic and real edge cases, and most of it never becomes a test. That is a strange thing to waste, and it is why I find more value in turning captured production behaviour into regression cases than in inventing synthetic ones from scratch,” Golikov said.
What is missing is visibility, clearly
He underlines the whole situation by explaining that what links these three is not a missing tool. The tools exist. What is missing is visibility: each failure is real, and each one has found a way to happen without making a sound.
So the practitioner question he would put to any team that wants to talk about quality is not “how many tests do you have” – it is “which of your failures are loud, and which are silent” because the sheer number of tests tells us almost nothing.
A thousand tests that never run, or that flip too often to trust, or that ignore the behaviour your users actually hit, is not a thousand units of safety. The honest measure of a suite is how reliably a real problem turns into a signal you cannot miss.
It’s a dirty, unglamorous job, but someone’s got to do it
According to Golikov, “Making failures loud is unglamorous work, and that is part of why it gets skipped. It means taking the checks that already exist and putting them where continuous integration (CI) will run them on every push. It means treating flakiness as a defect in its own right, quarantining the offenders so they stop poisoning trust, and fixing the underlying race rather than re-running until green. It means looking at what your logs say the service does and asking which of that you would actually notice breaking. None of it is clever. All of it is the difference between a suite that protects you and one that just makes you feel protected.”
Talking about the practicalities here and drawing from his own experiences at the command line, Golikov says that, personally, he builds small open source tools around these gaps, but the tools are the easy part. The hard part, the part no script does for you, is the decision to make a failure impossible to ignore, and then to keep it that way when the next deadline arrives and a red build is inconvenient.
Don’t count the tests, find the failures
The bottom line here?
“Quality is not the absence of failures,” Golikov concludes. “It is the guarantee that the failures you have cannot stay quiet. If you want to know how healthy a codebase really is, do not count its tests. Find the failures it is currently capable of having without telling anyone. Then make them loud.”
Mikhail Golikov builds open source Python testing tools under the handle golikovichev.