Deterministic tests
Section titled “Deterministic tests”A deterministic regression test gives the same result when the relevant code, test, inputs, and controlled environment are unchanged.
A test that sometimes passes and sometimes fails without a relevant change is flaky. Flakiness weakens the value of a failure signal because the team cannot immediately trust the result.
Problem
Section titled “Problem”Automated regression tests are useful when a failure provides credible evidence that something changed or broke.
Non-deterministic failures introduce noise. Teams start rerunning tests, ignoring known flakes, or treating a red build as provisional.
Sources of non-determinism
Section titled “Sources of non-determinism”Common causes include:
- shared mutable state between tests;
- dependence on execution order;
- real clock time, time zones, or timing windows;
- random input without a recorded seed;
- asynchronous work observed through fixed sleeps;
- remote services or unstable networks;
- unisolated files, ports, databases, or global process state;
- resource leaks and incomplete cleanup;
- concurrency races in the test or system under test.
The correct fix depends on which input or state is uncontrolled.
Control the relevant inputs
Section titled “Control the relevant inputs”A deterministic test should make important inputs explicit or reproducible.
Useful techniques can include:
- create a known fixture for each test or isolated test group;
- inject or control clocks when wall time is not the behavior under test;
- record random seeds so a failure can be reproduced;
- wait on observable completion conditions instead of arbitrary sleeps;
- isolate filesystem, database, and network state where practical;
- replace an unrelated unstable remote dependency with a suitable test double;
- make cleanup failures visible instead of silently contaminating later tests.
Do not remove realism that is required for the risk being tested. A test can be deterministic while still using real infrastructure if that infrastructure is controlled enough for the test purpose.
What deterministic does not mean
Section titled “What deterministic does not mean”Deterministic does not mean that all tests must use fixed literal data or avoid concurrency.
Property-based and randomized tests can be useful when failures preserve the seed or generated case needed for reproduction.
Performance tests also measure values that naturally vary. Their assertions need statistical or threshold semantics rather than pretending every run is identical.
The key requirement for regression tests is that the pass or fail decision has a controlled explanation.
Failure modes
Section titled “Failure modes”Common problems include:
- adding longer sleeps until an asynchronous test usually passes;
- retrying every failed test and hiding the original failure rate;
- allowing tests to depend on state left by earlier tests;
- using the current date or time without controlling boundary cases;
- quarantining flaky tests indefinitely;
- replacing important real boundaries only to make the suite green.
Maintenance and execution cost
Section titled “Maintenance and execution cost”Isolation and reproducibility can require additional fixture setup, test seams, containers, or controlled environments.
These costs should be compared with the cost of unreliable feedback. A fast suite that frequently produces false failures can consume more engineering time than a slightly slower deterministic suite.
When a test becomes flaky, preserve evidence about the failure and fix the uncontrolled condition. Quarantine can limit damage temporarily, but it should not become the permanent state of valuable regression coverage.
Sources
Section titled “Sources”- Martin Fowler. “Eradicating Non-Determinism in Tests.” 2011.
- Celal Ziftci and Diego Cavalcanti. “De-Flake Your Tests: Automatically Locating Root Causes of Flaky Tests in Code At Google.” ICSME, 2020.