QA testing with a CLI agent
Smoke-test your app from Claude Code or Codex CLI with webagent, from first connection to a result you can share.
You have a staging site, a test login and a terminal with Claude Code or Codex CLI. This section gets you from there to a clear pass or fail for a flow like login or checkout.
Webagent gives your agent cloud browsers over MCP. The agent calls smoke_test with your page and what you expect. Webagent opens a recorded browser, runs the test and returns a report with one verdict. Your agent reads it and tells you what passed, what failed and where.
The flow
Get an API key
Create a Member key in the dashboard under Workspace, API keys. Steps are in the Quickstart. Keep the key in an environment variable, not in a file in git.
Connect your agent
Add <API_URL>/mcp as an MCP server with your key as a Bearer token. One command or one config block:
Check that the agent lists the webagent tools. There are 15 fixed tools, plus one skill_<name> tool for each ready-made check your workspace can run.
Give the agent its instructions
Your agent does not know webagent's tools until you tell it. Paste the short snippet from Agent setup into your AGENTS.md (Codex) or CLAUDE.md (Claude Code). It says which tool to use, how to poll for the result, and what to put in the report.
Point the agent at a reachable app
The cloud browser opens public addresses only. Use a staging or preview URL. If your app runs on localhost, the agent starts a tunnel so the browser can reach it. See the local apps guide.
Use a test account. Do not give the agent a real person's credentials.
Ask for a smoke test
Describe the page or flow and what must be true afterwards, in one sentence. For example:
Smoke test https://staging.example.com/login. Sign in with the test user
[email protected] and the password I give you now, and check that the page shows
"Welcome" and the address contains /dashboard. Give me the verdict, the failed
steps, console errors and failed network calls, and the report link.The agent calls smoke_test once. It records the browser, runs the steps and the checks, and waits up to 60 seconds. It answers with a report. If the test is still running, the verdict is running and the agent calls get_report with the id until it ends.
More prompts to copy are in Recipes.
Read the result
The report has a verdict (pass, fail or error), the steps, console errors, failed network calls and links to the proof. Reading results explains every field, the difference between a failed test (fail) and a test that could not run (error), and how long the links last.
Share the evidence
While a session is open, you can watch it live in the dashboard under Sessions. Every smoke_test is recorded and its report has a screenshot of each failed step. See Evidence for what exists and how long it is kept.
Two ways to run a test
Test (smoke_test) | Goal task (run_task) | |
|---|---|---|
| You give | A URL, plain-language steps and what to expect: expectText, expectUrl, expectNoText | A goal in plain language |
| Best for | "Smoke test my app": a page or a flow with a clear pass or fail | Exploring, or a flow where the agent must decide what to do |
| Checks | Real assertions, links, console errors, failed network calls | None. ok means the agent says it is done |
| Comes back with | A report: verdict, steps, findings, console and network errors, evidence links | Answer, steps, reason it stopped. Call get_report for a report |
| Records | Always | Only with record: true |
| Waits | Up to 60 s by default, at most 90 s | Up to 90 s per call, then keeps running |
A test that outlasts the wait comes back with verdict running. The agent calls get_report until it ends. The agent setup snippet tells it to do that.
Known limits today
- The browser cannot open
localhostor private addresses. Use a public URL or a tunnel. - A goal task has no pass or fail assertions of its own.
okand a reportpassmean the agent says it is done. For a check, usesmoke_testwithexpectText,expectUrlorexpectNoText. smoke_testcannot use a saved login or the credential vault yet. For a login flow, write the login as plain-languagestepswith a throwaway test account (the step text is stored with the run), or userun_taskwithcredentials. See Signed-in sites.- Broken links and accessibility are checked on the start page only, not on pages your steps reach.
isErrormeans the test could not run, not that your app has a problem. A failed test is a normal result with verdictfail.- Console errors and failed network calls come from the session log, which is saved when the session ends. A report read right away can say the log is not available. Read it again.
- Evidence links expire. A report link works for minutes, not days.