> ## Documentation Index > Fetch the complete documentation index at: https://docs.erdo.ai/llms.txt > Use this file to discover all available pages before exploring further. # Experiments > Run structured tests — hypothesis, variants, decision rule — and decide whether a change worked. # Experiments An **Experiment** is a structured test: a hypothesis, one or more **variants**, a **decision rule**, a **primary metric**, and the **evidence datasets** that measure it. It's the "decide" side of an Erdo loop — a [Workstream](/workstreams) does the work, an Experiment decides whether a change worked. An experiment can be hosted by a workstream or stand alone. The day-to-day measurement is deterministic: a recipe reads metrics and appends **observations** (metric reads, decision checks, actions) to an append-only log — no agent per tick. You read that log to see whether the loop is working, then record the **decision** (ship / stop / iterate / inconclusive). In your workspace, Experiments live under **Activity**, usually inside the [Workstream](/workstreams) that hosts the work. An agent sets one up with a hypothesis and a decision rule, the measurement loop appends observations over time, and you read the evidence and record the call. They can also stand alone when you just want to test one thing. ## Lifecycle `planned` → `running` → `reading` → `decided`. When you decide, set `status=decided`, a `decision` (`ship`/`stop`/`iterate`/`inconclusive`), and an `outcome_markdown` narrative together. A new experiment starts in `planned` and is **inert** — it measures nothing until you **arm the measurement loop**. Arming has two steps: configure a scheduled measurement recipe (a `script_js` dataset refresh that calls `experiment.recordObservation` per variant, with `experiment_run_id` in its parameters), then flip it to `running`. An agent that creates an experiment does both in the same turn; if you create one via the API/CLI directly, do the same so it doesn't sit silently recording nothing. Measurement needs each variant to be identifiable in the evidence data (e.g. one ad group per variant, or a landing-page path) so observations can be attributed. You don't write a measurement recipe per experiment. A **generic, config-driven recipe** (`recipe-experiment-measure`) is the default: each variant declares how to filter its own rows per dataset (`{ column, values }`) and each metric declares the SQL that computes it (returning a `value` column, with `{{variant_filter}}` and `{{window}}` placeholders). The recipe loops metrics × variants and records the observations — no per-use-case code. ## Observation types — measurements, forecasts, and orderings The observation log holds more than metric reads. Three types matter when you're reading whether the loop is working: * **`metric_read`** — what reality measured for a variant (cost per lead, conversion rate). This is the ground truth every other type is scored against. * **`prediction`** — a *forecast* a judge made before reality spoke: "I expect variant B to convert better." A judge is any cheaper-than-reality opinion — a critic lens, a persona panel, a pairwise tournament, or a human's answer to a [choice](/attention). A prediction is only worth something once a later `metric_read` on the same variant lets you check it, so predictions are stored so they can be *paired* with the measurement that eventually arrives. * **`comparison`** — an *ordering* between variants rather than a value: "B beat control." Two predictions imply an ordering, an explicit head-to-head records one directly, and a human [choice](/attention) tied to a running experiment writes one every time you pick — your pick marks the chosen variant as beating the ones you didn't. You read them the same way as any observation — `erdo experiment observations --type prediction` (or `comparison`) — and they carry the same append-only, RBAC'd guarantees as metric reads. ## Judges and the calibration record Predictions and comparisons are only useful if you find out how often they were *right*. That's what **calibration** is: pairing each judge's forecasts and orderings against the outcomes reality later measured, so a judge earns trust from its track record instead of assertion. The headline signal is **pairwise agreement** — of the variant pairs where both the judge and reality expressed an ordering, how often the judge ordered them the way reality did. It covers prediction rows (two forecasts imply an ordering) and comparison rows (an explicit ordering) uniformly, against reality's ordering from the measured metric reads. Alongside it you get raw prediction and comparison counts, how many predictions reality has caught up with (*paired predictions*), and — because a human choice is just another judge — the same reading for your own answers. The calibration readout is the trust dial: it's a **query over the ledger**, not a separate subsystem, so it stays honest as more observations land. A [landing-page A/B test](/ab-testing-pages) is the fullest example: the automatic critics predict which page will convert, your [attention-feed choices](/attention) record human orderings, and the lead dataset supplies the `metric_read` that scores both — so over time you learn which judge to trust before you've spent the traffic to be sure. **Default judges are platform-calibrated, so they aren't blind on day one.** A new workspace has no paired outcomes for weeks, which would leave the default lenses Erdo ships with no track record. For those default lenses only, the calibration prior pools *statistics* across every workspace — agreement rates and sample counts, never anyone's page content, verdicts, or identity — so a default judge starts from the whole platform's record. That prior is then adjusted by your own paired outcomes as they accumulate: with enough of your own results a judge whose reality diverges from the platform's overrides the shared prior. Judges you author stay private to your workspace and never pool. While a judge's local record is still thin, Erdo keeps a slightly larger share of bets on the random arm, so your own traffic confirms or refutes the shared prior faster. ## The judge profile page Every judge has a profile — open one from the **Judges** tab under **Activity**, or from an item's [actor chip](/attention). A judge is a **review lens** (a critic skill): its profile shows the **rubric** it grades against, the artifact **kind** it screens, and whether it's a **default** lens Erdo ships or one **generated** for your org. Default lenses carry bootstrap trust and may block a build before any calibration exists; a generated judge enters *untrusted* and has to earn screening power from its record. Below that is the judge's calibration, made legible without a query: its **pairwise agreement** and sample counts, a breakdown **by artifact kind**, and how many of its scored pairs came from the **random arm** — the bets it did *not* gate, the only place its screening value is directly measurable. The **misordered pairs** panel is the point: each row is a case where reality ranked two variants the opposite way the judge did — grounded numbers, not prose. This is what answers *"why did the critic block this?"* without touching SQL, and it's also the seed the engine uses to generate a sharper judge. A **recent predictions** list rounds it out with the forecasts the judge has been making lately across every experiment you can see. ## The decision policy For **page experiments**, prose decision rules and fixed sample sizes are slow — our traffic takes weeks to reach a fixed N, and nothing moves traffic while the evidence accumulates. The **decision policy** replaces both. It runs as the measurement recipe's final step and does three things from the same per-variant counts the recipe already read: * **Belief.** Each variant gets a Beta-Binomial posterior on the primary metric's rate. If the [persona panel](/persona-panel) has a *calibrated* prediction for the variant, that prediction seeds the posterior as pseudo-sessions — but only once the calibration ledger holds enough paired experiments **and** its falsification test passes; until then the prediction carries zero weight and the belief is data-only. A calibrated prediction is capped so a real read of a few hundred sessions always dominates it. * **Allocation.** A constrained Thompson-sampling step nudges each variant's `allocation_percent` toward its posterior win-frequency, but never more than 15 points a day, never below a 10% floor while a variant is live, and never dropping the control below 20% until the experiment decides. The caps make it boring by construction: a bad prior or a bad day shifts spend modestly, never zeroes a variant out. A step that moves more than 30 points in total raises an [attention item](/attention). * **Stopping.** A posterior rule replaces the prose one: ship a variant when it's ≥ 95% likely to beat the control (with an expected-loss guard against winning by a hair on huge uncertainty), kill it when it's ≤ 5% likely, or declare inconclusive at a 28-day horizon. A guardrail whose posterior is > 90% likely to have regressed vetoes a ship. When the rule settles on ship or stop, it fires the same decision loop the prose path uses. The written outcome also reports a **prediction-augmented effect** (an HAIPW estimate) alongside the naive one — a tighter confidence interval when the prediction is good, and never a biased point estimate when it isn't. It's reported, not gating: decisions ride the posterior rule above. Read the policy's latest state — per-variant posteriors, current allocation, win frequencies, stop probabilities, the calibration gate, and the recommended decision — with `erdo experiment policy `, the `erdo_get_experiment_policy` MCP tool, or `GET /v1/experiments/:slug/policy`. It's also attached to the experiment GET response as `policy_state`. ## Programmatic access Everything below is **org-scoped and RBAC'd** (project *view* for reads, *contribute* for writes) and referenced by **slug**. ### CLI ```bash theme={null} # inspect erdo experiment list --status running erdo experiment get acme-cpl-ab erdo experiment observations acme-cpl-ab --type metric_read # the loop's evidence erdo experiment observations acme-cpl-ab --judge artifact-critic # one judge's forecasts here erdo experiment policy acme-cpl-ab # decision-policy state (page experiments) # create + drive — variants, evidence datasets and the decision rule in one call erdo experiment create \ --project acme \ --slug acme-cpl-ab \ --title "Landing page CPL A/B" \ --workstream acme-lead-engine-2026-06 \ --primary-metric cost_per_lead \ --hypothesis "Variant B's shorter form lifts conversion" \ --decision-rule "Ship B if its lead_submit_rate beats control by 2σ over 7 days" \ --guardrail bounce_rate \ --variant "key=control,label=Current page,control,page=" \ --variant "key=b,label=3-field mobile form,alloc=50,page=" \ --evidence "slug=acme.funnel-events,role=primary" \ --evidence "acme.leads:guardrail" erdo experiment set-status acme-cpl-ab running erdo experiment decide acme-cpl-ab --decision ship --outcome "B cut CPL 22% over 2 weeks, shipping" # the trust dial — each judge's paired predictions and pairwise agreement vs reality erdo experiment calibration --project acme erdo experiment calibration --judge artifact-critic # one judge's track record # rank the variants pairwise (comparisons persist to the ledger, so the judge earns calibration) erdo experiment tournament acme-cpl-ab --criteria "which page converts cold paid traffic best" \ --kind landing_page --max-comparisons 20 # record that you acted against the allocator's recommendation for one variant erdo experiment record-decision acme-cpl-ab \ --variant b --recommended hold --chosen kill --reason "client pulled the offer" ``` Each `--variant` is comma-separated `key=value` pairs plus the bare flag `control` (marks the baseline). `page=` is shorthand for `type=page` + that resource id — it links the variant to the page it serves. `alloc=` is the traffic split (0–100). Each `--evidence` is `slug=,role=primary` or the shorthand `:` (role defaults to `primary`). `tournament` requires `--criteria` and accepts optional `--kind`, repeated `--variant ` flags to restrict the field, and `--max-comparisons`; it runs synchronously and can take minutes. `record-decision` logs an allocation override — the variant, the allocator's `--recommended` action, the action you `--chosen`, and a `--reason` — and that disagreement becomes calibration data on the allocator itself. ### MCP tools | Tool | What it does | | ----------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------- | | `erdo_list_experiments` | List experiment runs (filter by workstream / scope / status). | | `erdo_get_experiment` | One experiment by slug — hypothesis, variants, evidence, decision. | | `erdo_get_experiment_policy` | The decision policy's state — posteriors, allocation, stop probabilities, calibration gate, recommendation. | | `erdo_create_experiment` | Start an experiment with variants, decision rule, primary metric, evidence datasets. | | `erdo_update_experiment` | Flip status or record the decision + outcome. | | `erdo_list_experiment_observations` | Read the append-only observation log — metric reads, predictions, comparisons, decisions. | | `erdo_judge_calibration` | Each judge's track record — paired predictions and pairwise agreement vs measured reality (the trust dial). | | `erdo_run_pairwise_tournament` | Rank an experiment's variants by pairwise comparison (Bradley–Terry); comparisons persist to the ledger so the judge earns calibration. | | `erdo_record_allocation_decision` | Record that you acted against the allocator's recommendation for a variant — the disagreement calibrates the allocator. | ### REST Base URL `https://api.erdo.ai`. `Authorization: Bearer ` + `X-Organization-ID`. | Method | Path | | ------- | -------------------------------------------- | | `GET` | `/v1/experiments` | | `GET` | `/v1/experiments/:slug` | | `GET` | `/v1/experiments/:slug/policy` | | `POST` | `/v1/experiments` | | `PATCH` | `/v1/experiments/:slug` | | `GET` | `/v1/experiments/:slug/observations` | | `POST` | `/v1/experiments/:slug/tournament` | | `POST` | `/v1/experiments/:slug/allocation-decisions` | | `GET` | `/v1/judge-calibration` | `/v1/experiments/:slug/tournament` requires `criteria` in the body and accepts optional `kind`, `variant_keys`, and `max_comparisons`; it responds synchronously with the ranking (can take minutes). `/v1/experiments/:slug/allocation-decisions` takes `variant_key`, `recommended_action`, `chosen_action`, and `reason`. `/v1/experiments/:slug/observations` takes optional `variant_key`, `observation_type`, `judge_slug` (filter to one judge's forecasts within the experiment), and `limit` query parameters. `/v1/judge-calibration` takes optional `project_slug` (limit to one project; omit to span every project you can view) and `limit` (max ledger rows scanned) query parameters.