Protected sites

Many staging sites refuse visitors they do not know. A WAF returns 403. Vercel asks for a sign-in. Cloudflare Access shows a login screen. An ngrok free tunnel shows a warning page. The agent sees that gate, not your app.

If you own the site, you can let webagent through with a secret header. You set the secret on your own site. You send the same value from the session. Our browser adds it to every request to the hosts you name, and to no other host.

The headers optionCopy Anchor Link

headers is an option of the session. Send it to run_task or create_session (MCP), or in the body of POST /v1/sessions (REST). A smoke test takes it in its session argument.

{
  "startUrl": "https://staging.example.com",
  "headers": [
    {
      "hosts": ["staging.example.com"],
      "headers": { "x-qa-bypass": "<your-secret>" }
    }
  ]
}

Rules, from the API contract:

• hosts lists host names. The header goes to each host and to its subdomains. No wildcards. Use the exact host of the site, for example staging.example.com.
• A request to any other host, including a redirect target on another host, does not get the header.
• Header names use letters, digits and -. host, content-length, transfer-encoding and connection are refused.
• Limits: up to 20 rules, 50 hosts per rule, 20 headers per rule, 4096 characters per value.
• Values are write-only. Session info, logs and network recordings show the header names, never the values.

The value travels in your create request, so use a secret you made for testing. Rotate it if you think it leaked. Never reuse a production secret.

Where you can send itCopy Anchor Link

The one-call form for a goal task is run_task with startUrl and headers:

{
  "goal": "Open the home page and tell me its title",
  "startUrl": "https://staging.example.com",
  "headers": [
    { "hosts": ["staging.example.com"], "headers": { "x-qa-bypass": "<your-secret>" } }
  ]
}

The tool opens the session with the header, runs the task and closes the session. For a smoke test, put the rule in session:

{
  "url": "https://staging.example.com",
  "expectText": "Welcome",
  "session": {
    "headers": [
      { "hosts": ["staging.example.com"], "headers": { "x-qa-bypass": "<your-secret>" } }
    ]
  }
}

If several tasks must share one browser, create the session first and run the tasks with its sessionId. run_task refuses headers together with sessionId.

curl -s -X POST "$API_URL/v1/sessions" \
  -H "Authorization: Bearer $WEBAGENT_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "startUrl": "https://staging.example.com",
    "headers": [{ "hosts": ["staging.example.com"], "headers": { "x-qa-bypass": "'"$QA_BYPASS_SECRET"'" } }]
  }'

From Claude Code or Codex, ask for it in one sentence:

Use the webagent smoke_test tool on https://staging.example.com with a headers rule
for host staging.example.com that sets x-qa-bypass to the value in my message. The
text "Welcome" must be on the page.

Pasting a secret into a chat sends it to the model provider. If that matters to you, use a secret that only unlocks a staging copy, or put the call in a script and not in a prompt.

Test itCopy Anchor Link

After the session starts, run a small task: "Tell me the page title, and say whether you see a login page, a block page or the app." If the header is right you see your app. If you still see the gate, check the host name and the secret.

A gate that answers 401 or 403 does not fail session creation. The session starts, and the error shows on the page and in the session timeline.

Vendor recipesCopy Anchor Link

The header name and where to create the secret come from each vendor. Check their page for changes.

Vercel deployment protectionCopy Anchor Link

Vercel has a feature called Protection Bypass for Automation. It is available on all plans. In the project settings you create a secret. Send it as a header named x-vercel-protection-bypass.

{ "hosts": ["my-app-git-feature-team.vercel.app"], "headers": { "x-vercel-protection-bypass": "<secret from project settings>" } }

Use the host name of the deployment you test. See Vercel's Protection Bypass for Automation page. It also lists what the bypass does not override, such as active DDoS mitigation.

Cloudflare AccessCopy Anchor Link

Create a service token in Zero Trust, under Access controls, Service credentials. Cloudflare shows the client secret once. Add a policy with the action Service Auth. Without that action Access asks for a login. Send both values as headers:

{
  "hosts": ["staging.example.com"],
  "headers": {
    "CF-Access-Client-Id": "<client id>",
    "CF-Access-Client-Secret": "<client secret>"
  }
}

See Cloudflare's service tokens page.

Cloudflare WAF custom ruleCopy Anchor Link

If a WAF rule or Super Bot Fight Mode blocks the browser, add a custom rule on your zone. Match a header you choose, and set the action to Skip. For example, with the header x-qa-bypass:

any(http.request.headers["x-qa-bypass"][*] == "<your-secret>")

Skip can skip rate limiting rules, managed rules and Super Bot Fight Mode rules. It cannot skip Bot Fight Mode, the setting on the Free plan. If that setting blocks you, you have to change the setting or the plan. No header of ours changes that. See Skip action. Header names in rule expressions are lowercase.

ngrok browser warningCopy Anchor Link

The ngrok free plan shows a warning page in front of HTML pages. A request with a header named ngrok-skip-browser-warning skips it. The value can be anything.

{ "hosts": ["<your-tunnel-host>.ngrok-free.app"], "headers": { "ngrok-skip-browser-warning": "1" } }

Use the host from the URL ngrok printed. On the free plan it changes each time, so make the session after you start the tunnel. See Test a local app for the tunnel steps and ngrok's free plan limits.

We ran the ngrok recipe end to end: with the header the session saw the page, without it the agent stopped on ngrok's warning page. We have not run the Vercel and Cloudflare recipes against a live session.

What is not available yetCopy Anchor Link

• A built-in verified agent. Today you set your own header, as above. Stored per-organization bypass headers and signed requests are not available. The options siteAccess and botAuth on a session are refused with invalid_input ("not supported yet"). We give no date.
• IP allow lists. Our browsers run on shared servers. We do not publish fixed IP addresses.
• Other WAFs. There are no step-by-step recipes for AWS WAF or Akamai yet. Any gateway that can match a request header and let it through works the same way.