A flaky test is an automated test that produces inconsistent results without any change to the code it covers: run it once and it passes, run it again and it fails, with identical code, test logic, and environment each time. That unreliability has a real cost, with developers losing roughly 2% of coding time to flaky test investigation and QA teams burning a large share of capacity on maintenance instead of expanding coverage. This guide covers what causes flaky tests in mobile apps, how to detect and fix them, and how Minitap's autonomous QA agent removes the root cause of flakiness by reading the running app directly instead of relying on selectors that break.
TLDR:
- Flaky tests produce inconsistent results across identical runs due to timing, environment state, or test ordering, not actual code failures.
- A 50-person team loses roughly $120,000 annually to flaky test investigation, with 40% of QA capacity consumed by maintenance.
- Most mobile flakiness stems from missing waits on async operations, test order dependencies, and environment inconsistency.
- Retries hide the underlying issue without fixing it, creating false confidence and masking real regressions.
- Minitap's AI reads the running app directly and adapts when UI changes, removing selector breakage that causes most script-based test flakiness.
What Are Flaky Tests?
A flaky test is an automated test that produces inconsistent results without any changes to the code it covers. Run it once, it passes. Run it again, it fails. Run it a third time, it passes again. The code, the test logic, and the environment are all identical across runs. The outcome is not.
The term comes from the informal meaning of "flaky" as unreliable or inconsistently behaving. In testing contexts, a flaky test case is one where the result cannot be trusted as a definitive signal about whether the software works.
There are a few concrete ways this surfaces in practice:
- A test passes in CI but fails when a developer runs it locally against the same build, with no configuration difference that explains the gap.
- A test fails on the first run of a suite, but passes when re-run in isolation without touching any code.
- A test passes consistently for weeks, then starts failing intermittently after a dependency update that has no apparent relationship to what the test covers.
Flaky tests are distinct from genuinely broken tests. A broken test fails because something in the product is actually wrong. A flaky test fails due to factors outside the logic being tested: timing assumptions, environment state, network latency, animation completion, or test ordering effects. That distinction matters because the response is different. A broken test tells you to fix the code. A flaky test tells you the test itself has a reliability problem worth investigating.
The Business Cost of Flaky Tests
Google's own data put the financial reality plainly. Developers spend 2% of coding time investigating flaky tests. For a 50-person engineering team at a $120,000 average salary, that 2% works out to roughly $120,000 in lost productivity annually (50 engineers × $120,000 × 0.02). That figure doesn't account for the QA maintenance burden from flakiness, which pulls a large share of capacity away from coverage that actually grows.
The interruption cost compounds both numbers. When a flaky failure pulls a developer away mid-task, research shows it takes 15 to 25 minutes to return to deep focus. A single false failure can cost 45 minutes per occurrence once you factor in investigation time, the context switch, and the ramp back. Across a team running daily CI, the hours disappear fast.
Root Causes of Flaky Tests in Mobile Apps
Flaky tests in mobile rarely stem from a single cause. The failure surfaces in your CI pipeline, but the origin is usually several layers deeper. Here are the most common culprits:
| Root Cause | How to Detect | How to Fix |
|---|---|---|
| Timing and async operations | Monitor test duration variance across runs; review screenshots captured mid-animation | Replace fixed sleeps with condition-based waits that poll until elements exist or network requests resolve |
| Environment inconsistency | Track failures that only appear after extended app runtime or under low-memory conditions | Reset databases, clear sessions, and tear down mocks between runs; test in environments that mirror production state |
| Test order dependency | Run test in isolation vs. full suite; flag tests that pass alone but fail in suite | Isolate test state so no test inherits side effects; avoid shared global state across tests |
| Third-party dependencies | Review logs for network errors or timeout failures at assertion time | Stub external dependencies so network variance stops determining test outcomes |
- Timing and async operations: Mobile apps rely heavily on async calls, animations, and background threads. A test that doesn't wait for the UI to settle will fail intermittently even when the app is working correctly.
- Environment inconsistency: A test passing in a clean environment may fail when the app has been running for hours with low memory and other processes competing for resources.
- Test order dependency: Tests that share mutable state or rely on a specific execution sequence will break unpredictably when that order changes, which happens often in parallel test runs.
- Third-party dependencies: Network calls, push notification services, and external APIs introduce latency and failure rates your test suite didn't account for.
How Flaky Tests Erode Team Trust and Velocity
When a test fails, the first question should be "what broke in the product?" When that question becomes "is this even a real failure?", something has gone wrong with the suite itself.
Flaky tests shift engineer behavior in ways that compound over time. Teams start re-running failures before investigating them. Failing tests get quarantined instead of fixed. CI pipelines develop a reputation for noise, so failures carry less weight. Each of these responses is rational in isolation, but together they hollow out the suite's signal value.
The velocity cost is real too. A developer waiting on a re-run to confirm a pass is burning time that should go toward shipping. Multiply that across a team and a sprint, and the drag adds up fast.
- Re-runs delay merge queues and extend release cycles without any corresponding product benefit.
- Quarantine lists grow quietly, covering failures the team has stopped investigating entirely.
- New engineers learn to distrust the suite early, and that instinct is hard to reverse once it forms. No amount of added manual QA process overhead changes that.
The trust problem is harder to measure than the time cost, but it matters more in the long run. A suite that engineers believe in gets run more often, gets maintained more carefully, and catches more real bugs before they ship.
Detecting Flaky Tests Before They Multiply
Early detection keeps a flaky test from becoming five flaky tests. The longer an unstable test stays in the suite, the more confidence erodes across the whole pipeline.
A few detection approaches worth knowing:
- Track pass/fail history per test across runs, not on a per-run basis. A test that fails once in ten runs looks fine in any single report but shows a clear pattern across runs.
- Flag tests that pass on retry without any code change. That retry success is the signal, not the failure itself.
- Monitor test duration variance. A test whose runtime swings wildly between runs often has a timing dependency before the assertion fails outright.
- Quarantine on threshold. Many teams pull a test from the blocking suite automatically after two or three non-deterministic failures, letting it run in a shadow suite while someone investigates.
Tooling That Helps
Most CI systems now have some form of flaky test detection built in. GitHub Actions, Buildkite, and CircleCI all surface retry patterns if you configure them to rerun on failure. Jest has a --detectOpenHandles flag that catches async leaks contributing to flakiness. Playwright's built-in reporter tracks flaky results across runs natively. Bazel marks tests with a flaky attribute that changes how the scheduler handles reruns.
The gap in most setups is that detection happens after the fact. You find out a test is flaky when it fails in CI, not before it gets merged. Shifting detection left, whether through static analysis, pre-merge test history checks, or AI-assisted anomaly detection on test output, moves the discovery window earlier and cuts the cost of fixing what you find.
Debugging and Fixing Flaky Tests
Reproducing the failure is step one. Run the test alone, then inside the full suite, to see whether the flake is self-contained or order-dependent. Many failures only appear when a prior test leaves shared state behind, so isolation tells you which problem you're actually solving.
From there, review logs and screenshots captured at the moment of failure. Timestamps reveal timing problems. A screenshot mid-animation means the test asserted before the UI settled. A network error in the logs means an external dependency wasn't ready when the test expected it to be.
In a 2020 IEEE study, Google developed a static analysis technique that automatically pinpoints root cause locations in flaky tests with 82% accuracy across 428 projects. Most failures trace to a small set of recognizable patterns: missing waits, unstable locators, and unguarded network calls. Once you know which category you're in, the fix is direct. Replace fixed sleeps with event-driven waits. Stabilize selectors so they bind to stable identifiers instead of positions that change. Stub external dependencies so network variance stops determining test outcomes.
The Quarantine Strategy for Managing Flaky Tests
Quarantine is a holding pattern. When a test flakes repeatedly, pull it from the required-check set so it stops blocking merges, but keep it running. The result still gets recorded. Coverage visibility stays intact, and other engineers stay unblocked.
The trap is treating quarantine as permanent. Minware's flaky test quarantine guide shows that without resolution criteria, quarantine lists grow indefinitely into technical debt. Track three numbers: total quarantined tests, median time in quarantine, and time to resolution. When those numbers climb without corresponding fixes, you have a backlog problem.
Test quarantine in practice works best with an explicit deadline: two weeks to fix or remove.
Why Retries Mask Problems Instead of Solving Them
Retrying a flaky test on failure is the most common response to flakiness, and also the one that causes the most long-term damage. The logic seems sound: the test failed, re-run it, it passed, move on. But that chain skips the most important question: why it failed in the first place.
A test that passes on the second attempt is still a flaky test. The retry just made the flakiness invisible in your reports. The underlying timing bug, the unguarded network call, the state leak from a prior test, none of those went anywhere.
The false confidence problem is the real danger. When retries are configured at the suite level, retries mask real failures from flaky ones, leaving teams unable to distinguish a suite that genuinely passes from one that passes only after absorbing failures silently., leaving teams unable to distinguish a suite that genuinely passes from one that passes only after absorbing failures silently. Over months, engineers stop asking whether the suite is reliable and start assuming it is.
Retries also create a coverage blind spot for real regressions. A genuine product failure that causes intermittent failures looks identical to a flaky one. If your retry config is set to three attempts, a real bug that fails on the first run but passes on the second gets classified as transient. The bug ships.
The compute cost is worth naming too. A suite with fifty flaky tests configured for three retries each runs up to 150 extra executions per pipeline, burning time and money on results that don't reflect actual product state.
Retries reduce the friction of dealing with flakiness day to day. They don't fix it.
Prevention: Writing Tests That Don't Flake
Waiting for async operations to finish is where most flaky tests are born. The fix is explicit waits over arbitrary sleeps.
- Use condition-based waits that poll until an element exists, is visible, or a network request resolves. A
waitForElementcall that polls every 100ms up to 5 seconds catches real-world timing variance without hardcoding a guess. - Isolate test state so no test inherits side effects from the one before it. Reset databases, clear sessions, and tear down mocks between runs.
- Avoid shared global state in tests. Two tests writing to the same in-memory cache or file path will race each other under parallel execution.
- Seed deterministic data. Tests that depend on live data or randomly generated inputs will produce inconsistent results by definition.
How Minitap Eliminates Flaky Tests Through AI-Powered Mobile QA
Script-based tools like Appium and XCUITest fail intermittently because selectors break when UI elements move. Minitap takes a different approach: the AI reads the running application directly and verifies whether user jobs complete successfully. When the UI changes, the agent adapts without anyone rewriting a selector.
That architectural shift removes the primary causes of flakiness at the source. Real-time monitoring of logs, CPU, and memory catches issues that scripted suites cannot reach. Minitap is a fully autonomous QA agent: the agent owns test authorship, maintenance, and root cause analysis, so your team writes no tests and investigates no failures. When something fails, a Slack alert arrives with a severity rating, a timestamped video, and a fix prompt you can paste directly into Cursor or another AI coding tool. Engineers never touch QA again.
Final Thoughts on Fixing Flaky Tests
When a test fails, the first question should be whether the product broke, not whether the test can be trusted. Flaky tests flip that equation, and once engineers learn to distrust the suite, real failures start getting ignored. You can fix the root causes with better waits and cleaner state management, but script-based automation will keep producing flakiness as long as your UI keeps changing. Minitap solves this by reading the running app and adapting to UI changes without requiring your team to rewrite selectors or own the test infrastructure.
FAQ
What is a flaky test?
A flaky test is an automated test that produces inconsistent results without any changes to the code it covers. It passes on one run, fails on the next, and passes again on a third run with identical code, test logic, and environment. The term comes from the informal meaning of "flaky" as unreliable or unpredictable.
Flaky tests fail due to factors outside the logic being tested, such as timing assumptions, environment state, or test ordering effects. That makes them distinct from broken tests, which fail because something in the product is actually wrong.
Best way to handle flaky tests without retries?
Track pass/fail history per test across runs to catch patterns, quarantine tests that fail non-deterministically after two or three occurrences, and investigate root cause immediately instead of letting them accumulate. Replace fixed sleeps with condition-based waits that poll until elements exist or network requests resolve, isolate test state so no test inherits side effects from prior runs, and seed deterministic data instead of relying on live or randomly generated inputs. Those steps reduce flakiness in script-based suites, but the underlying maintenance burden stays with your team. Minitap's autonomous QA agent removes the problem at the architectural level: it reads the running app directly and adapts when the UI changes, so selector breakage, the primary cause of script-based flakiness, never surfaces in the first place.
What causes flaky tests in mobile apps?
Timing and async operations are the primary cause: tests that don't wait for UI to settle or animations to complete will fail intermittently even when the app works correctly. Environment inconsistency, test order dependency from shared mutable state, third-party dependencies introducing latency, and device/OS variation across gesture recognition and display timing all contribute. A test passing in a clean environment may fail when the app has been running for hours with low memory and other processes competing for resources.
Selenium flaky tests vs Playwright flaky tests?
Both suffer from selector brittleness when UI elements move. Playwright's built-in reporter tracks flaky results across runs natively, while Selenium relies on third-party plugins like the Flaky Test Handler to surface retry patterns. Neither removes the root causes: timing dependencies, unstable locators, and unguarded network calls stay baked into script-based test suites no matter how detection is configured. Minitap's autonomous QA agent removes those root causes at the architectural level by reading the running app directly and adapting to UI changes without selectors, eliminating the dependency that makes both Selenium and Playwright suites brittle as products evolve.
Can AI-powered mobile QA eliminate flaky tests entirely?
Yes, by removing the root cause at the architectural level. Minitap reads the running application directly and verifies whether user jobs complete successfully, so when the UI changes, the agent adapts without anyone rewriting a selector. Script-based tools like Appium and XCUITest fail intermittently because selectors break when UI elements move. Minitap's approach removes that dependency entirely, and the device logs it captures surface issues that scripted suites cannot reach.