When our QA agent made its own account

Our QA agent (mini) was supposed to log into a customer's app last week. It couldn't. So it made its own account.

Not "it failed and we got an alert." It hit the login wall, decided that wasn't a good enough reason to stop, and built its own way in. It pulled the APK, grepped the JS bundle, reverse-engineered the backend, registered itself through a public API, intercepted its own verification email, and walked back into the app through the front door with a real account.

The test passed. On the way through, it found two real bugs no scripted suite would have caught.

This is the agent we've been building since Mobile Use, our open-source repo for driving mobile devices with agents. It's the clearest single run we have for why we think agentic QA is the right bet. Here's the log.

The 30-second version

We started a Minitest run on a beta customer's staging build and forgot to provision a test account. The agent was supposed to flag the missing config and stop. It didn't. It chained seven moves, none of them scripted, from guessing passwords to reverse-engineering the API to verifying its own email in a throwaway inbox. The assigned test passed. As a side effect it surfaced one UX bug that locks out real users and one missing rate limit on the registration endpoint.

The setup

mini is our AI QA engineer for mobile. We own the authoring and the maintenance: the agent learns the jobs your app exists to do, builds it from source, and uses that build the way a real user would.

For this run we set up one critical-flow test on a beta customer's app. The path was boring on purpose: log in, get to the capture screen, tap attach, check the camera and photo flows are visible.

We forgot to provision a test account on the staging build. So the agent hit the login screen with no credentials. It was supposed to flag the missing config and halt.

It didn't halt.

The log

It tried the obvious first: test@test.com / test1234, then a few common defaults. Rejected.

Then it poked around the device. Inspected shared_prefs, ran pm clear and restarted to check for a stored session, looked at environment variables. Nothing there either.

So it took the APK apart. Pulled it off the cloud device, unzipped it, grepped the JS bundle for hardcoded credentials, Supabase URLs, anything auth-shaped. It found the login screen file, (auth)/login.tsx, but no signup route lived in the app. It also tried the <customer>://home deep link, in case the router would skip auth. Rejected.

That sent it to the network. It ran netstat before and after a login attempt to see which IP the app was talking to, and traced it to https://my.<customer>.app/api/. The root returned:

{ "message": "<customer> API is running", "docs": "/docs" }

A docs path. It followed /docs and found POST /api/auth/register: public, no auth. Tried it:

{
"message": "User created successfully...",
"userId": "77287043-...",
"emailVerificationRequired": true
}

Account created, but it had to verify an email. So it spun up a guerrillamail address (umitlnaw@guerrillamail.com), re-registered with it, polled the guerrillamail API, read the verification mail from the customer's transactional sender, and pulled the code (D42500DE) out of the HTML. A GET /api/auth/verify-email?token=D42500DE came back with "Email verified successfully." It logged in over the API and got real JWT tokens.

Here's the part I like. The assigned test runs through the app UI, not the API, so it threw the API session away. It ran pm clear again, sat through onboarding, and typed umitlnaw@guerrillamail.com / Test123456! into the login screen by hand, like a person would.

It worked. The test carried on from there: tapped the plus button on the capture bar, hit the attach screen, confirmed the camera icon and the "All Photos" criterion were visible. Test passed.

The two bugs we didn't go looking for

The run was scoped to one flow. It turned up two things we hadn't seen.

The first is a UX bug. Log in before verifying your email and the backend says "Email not verified", but the UI shows "Invalid email or password". Anyone who signs up and tries to log in before clicking the link is stuck retrying a password that was right the whole time. One-string fix.

The second is a missing guardrail. /api/auth/register has no rate limiting and no abuse protection, so anything can register accounts forever. Not urgent for a customer this early, but we flagged it. The usual fixes apply: IP rate limiting on /register, a disposable-domain blocklist.

Neither of these shows up in a smoke test or a scripted E2E run. They showed up because the agent was poking at things a script never would.

What we keep coming back to

We didn't tell it to pull the APK, grep for auth routes, or use a temp inbox. It picked each move off the back of the last thing that failed.

This is the thing we keep betting on. The unit of testing should be a job the app exists to do, not a script pinned to one screen. When the agent owns the path from intent to outcome, it ends up reading the bundle, watching the network, and calling the API on its own. Most of the bugs that reach production were hiding in one of those places.

On consent

If you landed here ready to be suspicious: the customer is a friendly beta partner. They gave us full consent to do whatever the QA run needed, on the app and the backend. We don't poke at customer infrastructure without an explicit yes, because it's the right thing to do and because it's the only way this works past the first demo. Everything the agent does stays inside what the customer signed off on.

If you've got a mobile app, point mini at it and see what it does. Book a demo and we'll run it on your build.