The most valuable thing an agent did for me was not code

The most valuable thing an agent did for me was not code.

Someone reported that an offline-capable app did not start properly without a network connection. That was weird. Offline use was one of the cases the app was built for.

So I let the agent work like a lab. It built the release APK, drove the emulator, toggled airplane mode and wrote a small proxy which killed only the API host while keeping the rest of the network alive. Then it dumped the on-device cache and compared screenshots.

The first finding was not the actual reason. But it found the location of the problem in one run.

A compression function produced unpaired UTF-16 surrogates for the cache, while the actual store persisted UTF-8. The invalid surrogates disappeared and so did the cache.

The fix is four lines. The value was before it.

TL;DR

• I did not use the agent to write code or search for a possible reason → I used it for the complete diagnosis
• Give the agent a wide investigation scope without giving it an equally wide implementation scope
• Define repeatable evidence before it starts changing things
• Autonomous execution still needs a human deciding what counts as proof
• If you don’t want agents writing production code yet, let them reproduce, measure and disprove

I didn’t use the agent to write code or just search in the codebase for a possible reason. I used it for the complete diagnosis.

And once that worked, I wanted to know how far I could take the method.

Start with a measurement contract

The next problem was about performance. An action in a mobile app could take up to ten seconds under load.

My own suspicion pointed mainly at the native integration layer. That turned out to be only part of the story.

The useful part of my prompt was not the suspicion. It was the definition of evidence. I wanted Android, the real app, a local backend, a production-shaped dataset, idle and load scenarios, and the same measurements before and after a change.

My instruction was roughly: continue step by step until the run is complete, and save every result.

This gave the agent a wide investigation scope without giving it an equally wide implementation scope. It could build fixtures, add measurements, drive the app and disprove my hypothesis. It could not turn every interesting thing it found into a product change.

That boundary matters. Investigation benefits from freedom. Production code benefits from a very clear reason to exist.

What the first run found

The first issue was database contention during background work.

With a large local dataset, the background process started one complete job per selected group. The concurrency limit I expected to protect us did not apply because these requests used synchronous execute() calls.

The lookup itself was fast while the system was idle. Under active background work, about 79.7% of its time was spent waiting behind concurrent database work. Limiting that work to three jobs changed the lookup p95 from about 110 ms to 11 ms. The maximum fell from 157 ms to 14 ms.

That was already useful, but it did not explain all of the reported delay.

A physical device reported a successful hardware input while the first JavaScript handler sometimes ran seconds later. So I attached the device and asked the agent to go as deep as it did before.

The agent injected package-scoped test events through Android debugging tools. They followed the same native receiver and event path as the hardware input, but stopped before creating application data. This made 45 identical probes per run possible.

The hardware was not the problem. The native event emission took 0.21 ms. Outside a stall, the event reached JavaScript in 1 to 3 ms. During cache persistence it waited behind compressToUTF16, which blocked the JavaScript thread for an average of 8.22 seconds on the large fixture. Writing the result to storage needed about 21 ms.

With the existing serializer, 25 of 45 probes waited at least one second and the maximum was 8.06 seconds. With the same data and raw JSON, the p95 fell from 7.66 seconds to 5 ms. Not one probe waited a second.

That result did not come from reading the compression code and calling it expensive. The agent had to prove where the input was waiting, separate dataset shape from payload size, and run the same input through both implementations.

Repeat the method, not the conclusion

After the first run I created a handoff and started a second investigation in another mobile app.

The prompt was intentionally broad: take over the complete local end-to-end investigation, build whatever profiling and fixtures are needed, and stay local.

The important part was what the agent did not do. It did not treat the first result as proof for another app. It treated it as a hypothesis.

On a physical device, compressing a production-shaped 1 MB cache took about 2.66 seconds.

Separately, a system trace on a recent Android test target showed the JavaScript thread running on CPU for 98.41% of the measured compression interval. That ruled out storage I/O and lock waiting.

Still not enough.

I asked what a user would actually notice. Was this work effectively in the background, or did it block something people do? I asked for a deterministic difference in a real app flow, not only a theoretical serializer number.

That changed what counted as finished.

The agent built a local authenticated flow, made a real tap collide with the persistence window, and measured native touch to the JavaScript handler to the first visible frame.

These Android test targets are a separate evidence lane from the physical device measurement above.

Across an older and a newer Android target, compression added hundreds of milliseconds before the tap handler ran. On the older target the complete interaction took more than a second. Switching to raw JSON removed around 96% of the wait before the handler. The later rendering time barely changed on the newer target. The tap itself had been waiting behind compression.

The final product fix removed 12 lines across three files.

Again, the value was before it.

A low-risk place to start

If you don’t want to use agents for production code yet, this is a very low-risk place to start: let them reproduce, measure and disprove.

A wrong hypothesis costs you a rerun, not a rollback.

Take a bug you understand well. Ask the agent to run the real app, reproduce the behavior and show you exactly where its explanation comes from. Let it add temporary instrumentation. Let it drive the simulator or the browser while it checks the logs and console. Ask for the same scenario before and after the change.

And if it hits a limit, don’t stop at “the agent cannot do this.” What is actually missing? Is the model not capable enough? Is access missing? Does the app have no test data? Is there no way to observe the behavior?

In one investigation the agent could not test an external integration locally, so I asked it to build a mock server from the available protocol documentation. For the performance work I let it add a lot of logs and measurements because the code alone could not tell us the runtime behavior.

Think about what you would do next, then ask the agent to do it for you.

Autonomous does not mean hands-off

The two sessions delegated dozens of tasks across multiple agents and subagents. Together the agent runs added up to several hours of active work. Because they ran in parallel, that was not the same amount of wall-clock time.

The number sounds interesting. It was not the reason this worked.

The investigation had a measurement contract, and I kept changing that contract when the evidence was too weak. I asked for another check when a database conclusion looked suspicious. I rejected the first report because it showed numbers without explaining the cause. In the second run I pushed the agent from serializer time to visible product impact.

The agent also rejected its own bad evidence. It discarded truncated Logcat records, stale app bundles, mixed run boundaries and taps which missed the target. It kept physical-device and emulator claims separate. Both long runs were paused with handoffs and resumed from the same measurement boundary.

That is what autonomous meant here. The execution continued without me driving every command. The judgment did not disappear.

You need to keep your hands on the steering wheel. If I had only asked the agent to search for a reason for these regressions, it would never have found or proven all of this. Agents are becoming more capable, but also better at following instructions. They will not start an investigation like this by accident.

You need to come up with the idea. You need to decide what evidence connects the technical result to the product. And you need to challenge the first answer when it does not explain enough.

That is also why this method fits the orchestrator pattern. The agent can execute a huge amount of work, but somebody still owns the goal and decides what comes back into the product.

Give the agent broad room to investigate, build fixtures and disprove hypotheses. Keep the write boundary narrow. And keep your hands on the steering wheel.