Skip to main content

​
Evals

Erdo evals score an agent’s work against a rubric, with pass/score tracking over time. They run in production and are designed to be driven by an AI coding agent (e.g. Claude Code via MCP): change how something is generated, run the suite, read the per-criterion scores, then add, rewrite, or remove cases as the generation changes. Two kinds of suite:
  • Text suites — judge the agent’s text answer against the rubric (e.g. the data-question-answerer).
  • Artifact suites (evaluate_artifact: true) — judge the rendered page the agent builds (landing pages, dashboards, apps). Erdo publishes the page, screenshots it on desktop and mobile, drives it in a real browser (submits the lead form, advances the carousel, confirms the voice widget loads, checks the conversion analytics event), and scores the screenshots + interaction with a vision model. A page that looks fine but doesn’t capture the lead scores low.
Suites are referenced by slug. Each suite has cases (name, input brief, rubric of weighted criteria), a judge model, and a pass_threshold (0–5). Cases run through the same path a real user/API call hits — a thread, a message, and the normal agent invocation — so an eval tests the actual product flow, not a synthetic one. (External egress like email/integration writes is mocked during an eval run; everything else is real.) A case may add setup_messages: ordered turns run before input in the same thread, for multi-step flows — e.g. create a voice widget, then build the landing page wired to it. The judge scores the result of input.

​
Evaluators

A case can specify evaluators with two distinct roles, so an eval discriminates instead of passing everything:
  • llm_rubric = the score. An LLM scores the output (or rendered page) against the rubric — the real signal; the case score is its weighted verdict.
  • script = a gate, not a scorer. A JS evaluate(ctx) returning {score, passed, reasoning}, run deterministically (no LLM tokens). ctx = {input, output, artifact, data_store}. Use it for structural prerequisites (the lead form exists, the test lead actually landed in data_store). A clean gate adds no points; a failed gate hard-fails the case (structure missing = broken).
The case score comes from the llm_rubric (a failed gate forces it to 0); a script-only case scores on the gate. Any evaluator that errors fails the case — never a silent pass. Set evaluators via --evaluator (CLI) or the evaluators field (MCP); omit to use the rubric as a single llm_rubric.

​
CLI

For artifact suites the case input is the builder’s JSON input — artifact_kind (landing_page | dashboard | app) plus a description. An artifact suite builds real pages in whatever org the command resolves to. Once every case is judged, the run cleans up the pages it built — they are soft- deleted, so they stop appearing in listings and their public links stop working. This keeps a later run of the same brand from discovering a leftover and editing it instead of building fresh (which scores it zero). The cleaned-up pages stay referenced from the run results and are restorable (erdo pages restore <id>) if you want to inspect or keep one. The active org set by erdo org use is machine-global and a concurrent session can switch it, so erdo eval run refuses an artifact-building suite unless you pin the org explicitly — pass --org <id|slug> (or set ERDO_ORG):
Text-only suites don’t build artifacts and run without a pin, though pinning is still the safer habit for any scripted run. See pinning the org in automation.

​
MCP tools

Available on the MCP server; identity and permissions ride your token, so you only see your organization’s and global suites.

​
REST

Base URL https://api.erdo.ai, Authorization: Bearer YOUR_API_KEY. Artifact suites are expensive (they build + render + score real pages), so they’re excluded from the daily cron (cron_enabled: false) and run on demand.