# Smoke tests in CI

Source: https://webagent-docs-staging.lookover.ai/guides/ci

> Run a webagent smoke test after every deploy from a CI job with curl or the SDK, and fail the job when the test does not pass.

A smoke test in CI is three calls: start a run, wait for the report, set the exit code from its verdict. This page uses the `smoke-uat` skill. It loads your page, checks for console errors, failed network calls, broken links and basic accessibility problems, and can follow plain-language steps and check what is on the page afterwards.

## What you need [#what-you-need]

- A **public URL** of the app you just deployed. The cloud browser cannot open `localhost` or a private CI address. See [Test a local app](/guides/local-apps).
- An API key. Create one on the Keys page. The **member** role is enough to run skills. Make a separate key for CI, so you can revoke it alone.
- The key stored as a CI secret, here `WEBAGENT_API_KEY`. Never write it in the repository or print it in the log.
- The skills seeded on the deployment you use. If `smoke-uat` is missing, the start call fails with `not_found`. Ask your admin.

> **Note:** `$API_URL` in the examples is your API host, `https://api.us.webagent.lookover.ai` once your workspace is live there. If your team gave you another URL, use that.

## The three calls [#the-three-calls]

| Step | Call | Reply |
|---|---|---|
| Start | `POST $API_URL/v1/skills/smoke-uat/run` with `{"inputs": {"url": "..."}, "session": {"record": true}}` | `202` with `runId`, `sessionId`, `status: "running"` |
| Wait and read | `GET $API_URL/v1/skill-runs/{runId}/report.json` until `verdict` is not `running` | The [report](/qa-testing/reading-results#the-test-report): `verdict`, `summary`, `steps`, `findings`, `consoleErrors`, `networkFailures`, `evidence` |
| Keep | Open `evidence.reportSignedUrl` | The HTML report with screenshots of failed steps. The link works for 5 minutes. |

Authenticate with `Authorization: Bearer <key>` or `x-api-key: <key>`.

The `verdict` decides the exit code:

| `verdict` | Meaning | Exit code in the script below |
|---|---|---|
| `pass` | Every step and check passed. | 0 |
| `fail` | The test ran and found a problem. Read `summary` and the failed `steps`. | 1 |
| `error` | The test could not run to a result: the run was cancelled, denied a permission, or the browser failed. | 2 |
| `running` | Still going. Read again. | (waits) |

A `fail` is a finding about your app, or about the test. An `error` means the test could not finish. Keep the two apart in your CI so nobody chases a bug that is not there. The `outcome` field has the raw skill status (`passed`, `failed`, `error`, `cancelled`, `denied`).

`GET /v1/skill-runs/{runId}` is still there for the raw run record, and `GET /v1/skill-runs/{runId}/report` redirects to the HTML report.

Console errors and failed network calls come from the session log, which is saved when the session ends. A report read the moment the run ends can say in `notes` that the log is not available. The script reads the report once more after 5 seconds when `evidence.logsPath` is missing.

## A script with curl [#a-script-with-curl]

This script reads the report from `report.json`. It was run against a local API with a pass (exit 0), a failed expectation and a missing page (both exit 1). A failed download of the HTML report does not change the exit code.

```bash title="scripts/smoke.sh"
#!/usr/bin/env bash
# Needs: curl, jq. Env: WEBAGENT_API_KEY, TARGET_URL. Optional: API_URL, EXPECT_TEXT.
set -euo pipefail

API_URL="${API_URL:-https://api.us.webagent.lookover.ai}"
auth=(-H "Authorization: Bearer ${WEBAGENT_API_KEY}")

started=$(curl -sS --fail-with-body -X POST "$API_URL/v1/skills/smoke-uat/run" \
  "${auth[@]}" -H "Content-Type: application/json" \
  -d "$(jq -n --arg url "$TARGET_URL" --arg text "${EXPECT_TEXT:-}" \
        '{inputs: ({url: $url} + (if $text != "" then {expectText: $text} else {} end)), session: {record: true}}')")
run_id=$(jq -r .runId <<<"$started")
echo "run: $run_id"

read_report() { curl -sS --fail-with-body "$API_URL/v1/skill-runs/$run_id/report.json" "${auth[@]}"; }

# Wait up to 10 minutes for a verdict.
for _ in $(seq 1 120); do
  report=$(read_report)
  verdict=$(jq -r .verdict <<<"$report")
  [ "$verdict" != "running" ] && break
  sleep 5
done

# The session log is saved a moment after the run ends: read once more if it is not there yet.
if [ "$verdict" != "running" ] && [ "$(jq -r '.evidence.logsPath // empty' <<<"$report")" = "" ]; then
  sleep 5
  report=$(read_report)
fi

echo "verdict: $verdict"
jq -r '.summary' <<<"$report"
jq -r '.steps[] | select(.status == "fail") | "failed step \(.index + 1) \(.title): \(.error // "no reason given")"' <<<"$report"
jq -r '.consoleErrors[] | "console: \(.text)"' <<<"$report"
jq -r '.networkFailures[] | "network: \(.method // "") \(.url) \(.status // .error // "")"' <<<"$report"

# The report link lives for 5 minutes: save the file now. A failed download must not hide the verdict.
link=$(jq -r '.evidence.reportSignedUrl // empty' <<<"$report")
if [ -n "$link" ]; then
  curl -sS -o smoke-report.html "$link" || echo "could not save the HTML report" >&2
fi

case "$verdict" in
  pass) exit 0 ;;
  fail) exit 1 ;;
  running) echo "still running after 10 minutes" >&2; exit 3 ;;
  *) exit 2 ;;
esac
```

To add plain-language steps and what must be true afterwards, extend `inputs`. The script above sets `expectText` from `EXPECT_TEXT`. For the other checks, `expectUrl` and `expectNoText`, add them the same way:

```json
{
  "inputs": {
    "url": "https://staging.example.com/login",
    "steps": ["type \"qa-user@example.com\" into Email", "type the test password into Password", "click Sign in"],
    "expectText": "Welcome",
    "expectUrl": "/dashboard",
    "expectNoText": "Invalid password"
  }
}
```

> **Warning:** Step text is stored with the run. Use a throwaway test account, never a real password. Skill runs cannot use vault logins yet, see [Signed-in sites](/guides/signed-in-sites). A broken link on the start page fails the run. To report it as a finding only, add `"strictLinks": false` to `inputs`.

To retry the start call without making a second run, send an `Idempotency-Key` header. The same key and body replay the first answer.

## GitHub Actions [#github-actions]

```yaml title=".github/workflows/smoke.yml"
name: Smoke test
on:
  workflow_dispatch:
  push:
    branches: [main]

jobs:
  smoke:
    runs-on: ubuntu-latest
    timeout-minutes: 15
    steps:
      - uses: actions/checkout@v4
      - name: Run the smoke test
        run: bash scripts/smoke.sh
        env:
          WEBAGENT_API_KEY: ${{ secrets.WEBAGENT_API_KEY }}
          TARGET_URL: https://staging.example.com
      - name: Keep the report
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: smoke-report
          path: smoke-report.html
          if-no-files-found: ignore
```

Add the key under Settings, Secrets and variables, Actions, as `WEBAGENT_API_KEY`. Pass the URL of the deploy you want to test. The job fails when the script exits with a code other than 0.

If your CI starts the app inside the job, the app is on a private address. Either deploy it first to a public preview URL, or put a tunnel in front of it for the length of the job. A tunnel makes the app public for that time, so use test data only.

## With the SDK [#with-the-sdk]

The TypeScript SDK wraps the same calls. It is a private package today, so use it only if your team has access. See [SDK](/sdk).

```ts title="smoke.mjs"
import { WebagentClient } from "@lookover-ai/webagent-sdk";

const client = new WebagentClient({
  apiKey: process.env.WEBAGENT_API_KEY,
  // baseUrl: "https://...",  // set this when your API host is not the one of your key's region
});

const { runId } = await client.runSkill("smoke-uat", {
  inputs: { url: process.env.TARGET_URL, expectText: "Welcome" },
  session: { record: true },
});
await client.waitForSkillRun(runId, { timeoutMs: 600_000 });
// The session log is saved a moment after the run ends: wait before reading console and network errors.
await new Promise((r) => setTimeout(r, 5_000));
const report = await client.getSkillRunReport(runId);

console.log(report.verdict, report.summary);
for (const s of report.steps.filter((s) => s.status === "fail")) console.log("failed:", s.title, s.error);
for (const e of report.consoleErrors) console.log("console:", e.text);
for (const n of report.networkFailures) console.log("network:", n.method, n.url, n.status ?? n.error);
process.exit(report.verdict === "pass" ? 0 : report.verdict === "fail" ? 1 : 2);
```

`waitForSkillRun` polls every second by default and throws a `timeout` error when `timeoutMs` passes. The default is 5 minutes. `getSkillRunReport(runId)` returns the [report](/qa-testing/reading-results#the-test-report) from `GET /v1/skill-runs/:id/report.json`. For a task, use `getTaskReport(taskId)`. They need SDK 0.3.0.

## Tips [#tips]

- Run the test after the deploy has finished, against the new URL.
- Keep CI keys apart from your personal key. Revoke a CI key when a job log leaks it.
- Limit one run to one question. Several short runs are easier to read than one long one.
- A `fail` in CI is the moment to read `summary`, the failed steps and the saved report file.