Evals

Erdo evals score an agent’s work against a rubric, with pass/score tracking over time. They run in production and are designed to be driven by an AI coding agent (e.g. Claude Code via MCP): change how something is generated, run the suite, read the per-criterion scores, then add, rewrite, or remove cases as the generation changes. Two kinds of suite:

Text suites — judge the agent’s text answer against the rubric (e.g. the data-question-answerer).
Artifact suites (evaluate_artifact: true) — judge the rendered page the agent builds (landing pages, dashboards, apps). Erdo publishes the page, screenshots it on desktop and mobile, drives it in a real browser (submits the lead form, advances the carousel, confirms the voice widget loads, checks the conversion analytics event), and scores the screenshots + interaction with a vision model. A page that looks fine but doesn’t capture the lead scores low.

Suites are referenced by slug. Each suite has cases (name, input brief, rubric of weighted criteria), a judge model, and a pass_threshold greater than 0 and at most 5 (omission defaults to 3.5). Cases run through the same path a real user/API call hits — a thread, a message, and the normal agent invocation — so an eval tests the actual product flow, not a synthetic one. (External egress like email/integration writes is mocked during an eval run; everything else is real.) A case may add setup_messages: ordered turns run before input in the same thread, for multi-step flows — e.g. create a voice widget, then build the landing page wired to it. The judge scores the result of input.

Deployed-experience targets

Most cases invoke the suite’s agent. A case can instead set the closed target_type field to exercise an already-deployed experience. The seeded target suites are excluded from the agent cron; the dedicated deployment/nightly workflow runs them.

Seeded suite	`target_type`	What actually runs	Verdict and side effects
`deployed-pages-live`	`live_page_fleet`	Every public HTML page at canonical desktop/mobile routes and each active custom-domain route, plus a daily rotating sample of up to three canonical and three custom-domain URLs captured at desktop/mobile fold/full-page sizes	Direct deterministic fleet verdict: navigation, content-ready identity, visible text, route, overflow, runtime errors, first-party requests, and render-critical external assets must pass. Cross-origin background fetch/XHR/beacon failures remain in evidence but do not fail an otherwise healthy rendered page. Every sampled visual score must be at least 40, have no high-severity defect, and avoid a 20-point drop from the last accepted eval baseline. Baselines advance atomically only after the whole review passes. Creates a stored page-review/baseline record; does not interact with or mutate the pages. Hard deadline: 30 minutes.
`widget-fleet-live`	`widget_fleet_live`	Every active widget’s public config and real `/voice-widget.js` open-panel iframe at desktop/mobile, including exact enabled modality controls; provider-ready voice widgets also mint a normal allowed-origin ticket and prove allowlisted-origin rejection	Direct deterministic score from passed checks. Never connects the ticket, requests media, starts voice/video, or sends text, so it creates no conversation/session. Hard deadline: 20 minutes.
`widget-text-live`	`widget_text_canary`	A fresh turn through the real mobile loader, text input, send binding, response bubble, and public transport; then a continuity follow-up on the same returned session	Exactly four ordered persisted user/agent turns are required. The browser turn has hard limits of 40 seconds for chat, 6 seconds for authoritative persistence, and 47 seconds total; the follow-up limits are 30, 6, and 37 seconds. The real answers and persisted evidence then go to the suite’s normal rubric judge. Uses real LLM traffic and stores the canary conversation.
`widget-voice-live`	`widget_voice_canary`	First, the real mobile widget captures a deterministic spoken microphone fixture, uploads AudioWorklet PCM, renders provider STT and the causally later agent response, and schedules returned audio. Then a typed relay turn supplies a stable semantic prompt through the normal ticket, production Erdo relay, and real voice provider.	Requires the browser microphone/STT/response/playback chain plus valid relay audio and exact authoritative post-call transcripts for both sessions. The typed relay’s hard limits are 5 seconds ready, 15 seconds first audio, 20 seconds answer, 25 seconds complete, and 90 seconds persistence; the combined target is capped at 5 minutes. Uses provider credit, debits normal Erdo credit, and stores both canary conversations.
`widget-video-live`	`widget_video_canary`	The real 390×844 widget gets microphone/camera permission, starts a normal Daily video session, publishes local audio/video, and receives live remote avatar video	Requires the exact authoritative video-session transcript after browser close, including a user turn that preserves the stable spoken words “voice” and “canary” and a nonempty agent answer. Browser completion is capped at 80 seconds, persistence at 90 seconds, and the whole target at 3 minutes. Uses real Daily/provider traffic and stores the canary conversation.

Targets are deliberately not open-ended. A case cannot provide a URL, widget id, public key, organization id, or credential: the runner resolves trusted server-side state. Page and widget fleet targets are global staff operations. Text, voice, and video targets can resolve only the authenticated organization’s unique DB-designated canary. Creating, editing, or running any deployed target requires staff access.

Authoring setup and targets

setup_messages belongs to an ordinary agent case. target_type bypasses the suite agent and runs one of the five target executors above, so the two fields are mutually exclusive.

Operation	MCP / REST body fields	CLI
Create agent setup case	`"setup_messages":["..."]`	`eval case add ... --setup "..."`
Create target case	`"target_type":"widget_text_canary"`	`eval case add ... --target-type widget_text_canary`
Update without changing either	Omit `setup_messages`, `target_type`, `clear_setup`, and `clear_target`	Omit the four corresponding flags
Remove setup	`"clear_setup":true`	`eval case update ... --clear-setup`
Remove target	`"clear_target":true`	`eval case update ... --clear-target`
Switch setup to target	`"clear_setup":true,"target_type":"..."`	`eval case update ... --clear-setup --target-type ...`
Switch target to setup	`"clear_target":true,"setup_messages":["..."]`	`eval case update ... --clear-target --setup "..."`

Those bodies apply to erdo_add_eval_case / erdo_update_eval_case over MCP and POST / PUT /v1/evals/suites/{suiteSlug}/cases... over REST. On create, send only one of setup_messages or target_type. On update, omission preserves the stored field. clear_setup cannot be combined with setup_messages, and clear_target cannot be combined with target_type.

Evaluators

A case can specify evaluators with two distinct roles, so an eval discriminates instead of passing everything:

llm_rubric = the score. An LLM scores the output (or rendered page) against the rubric — the real signal; the case score is its weighted verdict.
script = a gate, not a scorer. A JS evaluate(ctx) returning {score, passed, reasoning}, run deterministically (no LLM tokens). ctx = {input, output, artifact, data_store}. Use it for structural prerequisites (the lead form exists, the test lead actually landed in data_store). A clean gate adds no points; a failed gate hard-fails the case (structure missing = broken).

The case score comes from the llm_rubric (a failed gate forces it to 0); a script-only case scores on the gate. Any evaluator that errors fails the case — never a silent pass. Set evaluators via --evaluator (CLI) or the evaluators field (MCP); omit to use the rubric as a single llm_rubric.

CLI

npm i -g @erdoai/cli

erdo auth login                           # paste an API key (multi-account, like gh)
erdo org list                             # your orgs (* = active)
erdo org use acme                         # set the active org (evals are org-scoped)

erdo eval suites                          # list suites
erdo eval create landing-variations --agent erdo.artifact-builder \
  --evaluate-artifact --no-cron \
  --case '{"name":"voice","input":"{\"artifact_kind\":\"landing_page\",\"description\":\"...\"}","rubric":[{"criterion":"voice widget loads","weight":2}]}'
erdo eval suite landing-variations        # show a suite + cases
erdo eval run landing-variations --watch --commit-sha "$(git rev-parse HEAD)"
                                             # poll to completion; non-zero on failure
erdo eval results <run-id>                # per-case scores + lenses
erdo eval runs --suite landing-variations # recent runs

erdo eval case add landing-variations \
  --name voice-concierge \
  --input '{"artifact_kind":"landing_page","description":"Landing page for ACME with a voice concierge ..."}' \
  --rubric '[{"criterion":"voice widget loads and is on-brand","weight":2},{"criterion":"hero + form render","weight":1}]'

erdo eval case add deployed-widget-checks \
  --name visitor-text-answer \
  --target-type widget_text_canary \
  --input 'What are your opening hours?' \
  --rubric '[{"criterion":"answers only from approved public knowledge","weight":1}]'

# Updates preserve setup/target unless you clear or replace it explicitly:
erdo eval case update deployed-widget-checks visitor-text-answer \
  --input 'Introduce yourself.' --clear-target --setup 'Prepare a concise answer.'
erdo eval case update deployed-widget-checks visitor-text-answer \
  --input 'Introduce yourself.' --clear-setup --target-type widget_text_canary
erdo eval case rm landing-variations voice-concierge

--target-type accepts only live_page_fleet, widget_fleet_live, widget_text_canary, widget_voice_canary, or widget_video_canary. For erdo eval create, put the same target_type field inside each --case JSON object. Text, voice, and video targets share one staff-designated canary in the pinned organization. Designation accepts a widget’s unique name or public key and resolves it only inside the authenticated organization. It requires an active text+voice+video widget, a provisioned provider agent, disabled scheduling, and a live provider configuration containing only the approved read-only knowledge and display-card tools. The runner revalidates the relevant modality and safety rules at execution time.

erdo --org acme eval canary set "Website Concierge"
erdo --org acme eval canary show
erdo --org acme eval canary clear

show returns configured: false when the organization has no designation. set and clear require --org (or ERDO_ORG) so another local session cannot switch the machine-global active organization underneath a staff mutation. The text target opens a fresh real session through the mobile widget’s visible input, sends the case input, reads the rendered user/agent bubbles, then reuses the returned session for a fixed continuity question grounded in the first answer. It passes only when the authoritative transcript contains exactly those four ordered user/agent turns and both rounds stay within their chat, persistence, and total latency limits. Evidence includes stored turn timestamps and any model/token/tool metrics present in the stored transcript; native text turns currently expose no provider metric fields, so none are fabricated. The voice target first opens the real mobile widget with greeting suppressed. A shipping-image microphone fixture with seven seconds of setup silence says “Hello Erdo”; the normal AudioWorklet uploads its PCM, provider STT must render that user turn, a later agent bubble must render, and later response audio must be scheduled. Because provider STT can legitimately spell the brand phonetically, the deterministic recognition gate requires the stable words “voice” and “canary” rather than an exact “Erdo” token. Its exact authoritative session transcript must preserve that user-before-agent order. It then sends the case input as a typed production-relay turn for stable semantic judging. All of these are hard acceptance requirements, not advisory metrics:

relay ready within 5 seconds, first audio within 15 seconds, agent transcript within 20 seconds, and the complete relay turn within 25 seconds;
authoritative webhook transcript persistence within 90 seconds and the entire browser-plus-relay target within 5 minutes (a 45-second relay timeout is the typed turn’s outer failure guard);
at least one audio frame containing even-aligned pcm_s16le at 16 kHz, at least 3,200 bytes / 100 ms of audio, and at least 160 non-silent samples (absolute amplitude greater than 64);
the same canary widget/session plus exact user and agent text in the authoritative stored transcript.

The video target starts the real mobile Daily call with scoped microphone/camera permission, publishes both local tracks, requires a live remote avatar video track, then closes the browser and reads the exact authoritative video transcript by the server-issued widget session id. Its browser, persistence, and total limits are 80 seconds, 90 seconds, and 3 minutes. Voice and video targets use normal provider usage and Erdo credit debit. There is no billing, provider, transcript, or public visitor test bypass.

Run source and production snapshot

Every run read/list response includes triggered_by and these optional revision fields: commit_sha (the primary revision for the suite), frontend_commit_sha, and backend_commit_sha. The admin run page links each stored revision to its immutable GitHub commit; erdo eval results prints full SHAs and erdo eval runs prints compact SHAs. All supplied SHAs must be full 40-character hexadecimal Git object names. Run requests accept source only for staff automation and only as post_deploy, nightly, or rollback. Without it, an admin-console run records triggered_by: "manual"; MCP/REST/CLI runs use the Knowledge surface and record knowledge; scheduled runs record cron. The deployed-experience workflow always records both frontend and backend snapshot revisions. Its primary commit_sha is the frontend revision for landing generation and deployed-page suites, and the backend revision for every widget-* suite, so DevBot links the failure to the code most likely responsible without losing the complete production snapshot.

erdo eval run widget-text-live --watch \
  --commit-sha "$BACKEND_SHA" \
  --frontend-commit-sha "$FRONTEND_SHA" \
  --backend-commit-sha "$BACKEND_SHA" \
  --source post_deploy

For artifact suites the case input is the builder’s JSON input — artifact_kind (landing_page | dashboard | app) plus a description. An artifact suite builds real pages in whatever org the command resolves to. Once every case is judged, the run cleans up the pages it built — they are soft- deleted, so they stop appearing in listings and their public links stop working. This keeps a later run of the same brand from discovering a leftover and editing it instead of building fresh (which scores it zero). The cleaned-up pages stay referenced from the run results and are restorable (erdo pages restore <id>) if you want to inspect or keep one. The active org set by erdo org use is machine-global and a concurrent session can switch it, so erdo eval run refuses an artifact-building suite unless you pin the org explicitly — pass --org <id|slug> (or set ERDO_ORG):

erdo --org acme eval run landing-variations --watch

Text-only suites don’t build artifacts and run without a pin, though pinning is still the safer habit for any scripted run. See pinning the org in automation.

MCP tools

Available on the MCP server; identity and permissions ride your token, so you only see your organization’s and global suites.

Tool	Purpose
`erdo_list_eval_suites`	List suites (slug, agent, settings)
`erdo_get_eval_suite`	A suite with all its cases
`erdo_create_eval_suite`	Create a suite + first cases (set `evaluate_artifact` for visual suites)
`erdo_update_eval_suite`	Update suite metadata, agent, threshold, artifact judging, or cron inclusion
`erdo_add_eval_case` / `erdo_update_eval_case` / `erdo_remove_eval_case`	Maintain cases; updates preserve omitted setup/target fields and accept `clear_setup` / `clear_target` for explicit removal or switching
`erdo_run_eval_suite`	Run a suite; accepts `commit_sha`, `frontend_commit_sha`, `backend_commit_sha`, and staff-only `source`, then returns `run_id`
`erdo_get_eval_run`	Run source/revision snapshot + per-case scores, criteria, reasoning, evidence, and artifact links
`erdo_list_eval_runs`	Recent runs with source/revision snapshot and aggregate pass counts/scores
`erdo_get_eval_widget_canary`	Show the current organization’s canary; returns `configured: false` when unset (staff)
`erdo_configure_eval_widget_canary`	Set by unique widget name/public key, or pass an empty `widget` to clear (staff)

REST

Base URL https://api.erdo.ai, Authorization: Bearer YOUR_API_KEY.

Method	Path
GET	`/v1/evals/suites`
GET	`/v1/evals/suites/{suiteSlug}`
POST	`/v1/evals/suites`
PUT	`/v1/evals/suites/{suiteSlug}`
POST	`/v1/evals/suites/{suiteSlug}/cases`
PUT	`/v1/evals/suites/{suiteSlug}/cases/{caseName}`
DELETE	`/v1/evals/suites/{suiteSlug}/cases/{caseName}`
POST	`/v1/evals/suites/{suiteSlug}/run`
GET	`/v1/evals/runs/{runID}`
GET	`/v1/evals/runs?suite_slug={slug}&limit={n}`
GET	`/v1/evals/widget-canary` (staff; current authenticated organization)
PUT	`/v1/evals/widget-canary` with `{"widget":"<unique name or public key>"}`; empty widget clears (staff)

The suite/case REST bodies use the same setup/target contract as MCP and the CLI. For example, an update that switches an existing setup case to the text canary is:

{
  "input": "Introduce yourself and explain how you help visitors.",
  "clear_setup": true,
  "target_type": "widget_text_canary",
  "rubric": [{"criterion": "Uses only approved public knowledge", "weight": 1}]
}

Switching back sends "clear_target": true plus setup_messages. Sending only clear_setup or clear_target removes that field. The incompatible pairs clear_setup + setup_messages and clear_target + target_type are rejected. The run body carries both attribution and the evaluated production snapshot:

{
  "concurrency": 4,
  "commit_sha": "1111111111111111111111111111111111111111",
  "frontend_commit_sha": "2222222222222222222222222222222222222222",
  "backend_commit_sha": "1111111111111111111111111111111111111111",
  "source": "post_deploy"
}

Canary configuration never accepts an organization id, URL, internal widget id, or auth override; organization and staff identity come from authentication. Artifact suites are expensive (they build + render + score real pages), so they’re excluded from the daily cron (cron_enabled: false) and run on demand.

Platform API

API Reference

Build apps

Python SDK

TypeScript SDK

Evals

Evals

Deployed-experience targets

Authoring setup and targets

Evaluators

CLI

Widget canary contract

Run source and production snapshot

MCP tools

REST

​Evals

​Deployed-experience targets

​Authoring setup and targets

​Evaluators

​CLI

​Widget canary contract

​Run source and production snapshot

​MCP tools

​REST

Evals

Deployed-experience targets

Authoring setup and targets

Evaluators

CLI

Widget canary contract

Run source and production snapshot

MCP tools

REST