HandyUI Agent

End-to-end testing of web apps is slow and brittle, and most automated testers fall over at the login wall: auth screens, one-time codes, and email verification steps. So the hardest flows stay manual.

I wanted an agent that could test a real app the way a person does, by looking at the screen and working through the login maze on its own.

It takes a written test plan, normalizes it into steps, and runs a durable orchestrator that drives a real cloud browser one step at a time. Each step is a small vision loop: perceive the page as an element inventory, text, and an annotated screenshot; plan a few actions with a vision model; act; then judge the result with a vision model. It gets through login by auto-creating and reusing test accounts, resolving email codes and magic links from a per-run inbox, and handling authenticator codes, while escalating the ones it genuinely can't clear, like SMS codes and CAPTCHAs, to a human who resolves and resumes the run.

It judges by looking at the rendered page rather than asserting on the DOM, which matches how a person decides whether a flow worked and survives markup changes. The orchestrator is durable and resumable with circuit breakers, so long runs can pause for a human and stop cleanly on auth loops instead of grinding.

I drew the human line deliberately: auto-resolve email and one-time codes, but escalate SMS and CAPTCHA, exactly where automation stops being reliable. Credentials are encrypted at rest with the keys kept out of prompts and logs, because it handles real login secrets.

Clicking buttons was never the hard part. Getting through the auth maze reliably, and knowing when to hand off to a human, was the whole problem.

A vision-based judge also turned out to be far more robust than DOM assertions for answering the only question that matters: did the flow actually work.