Skip to main content

​
Experiments

An Experiment is a structured test: a hypothesis, one or more variants, a decision rule, a primary metric, and the evidence datasets that measure it. It’s the “decide” side of an Erdo loop — a Workstream does the work, an Experiment decides whether a change worked. An experiment can be hosted by a workstream or stand alone. The day-to-day measurement is deterministic: a recipe reads metrics and appends observations (metric reads, decision checks, actions) to an append-only log — no agent per tick. You read that log to see whether the loop is working, then record the decision (ship / stop / iterate / inconclusive). In your workspace, Experiments live under Activity, usually inside the Workstream that hosts the work. An agent sets one up with a hypothesis and a decision rule, the measurement loop appends observations over time, and you read the evidence and record the call. They can also stand alone when you just want to test one thing.

​
Lifecycle

planned → running → reading → decided. When you decide, set status=decided, a decision (ship/stop/iterate/inconclusive), and an outcome_markdown narrative together. A new experiment starts in planned and is inert — it measures nothing until you arm the measurement loop. Arming has two steps: configure a scheduled measurement recipe (a script_js dataset refresh that calls experiment.recordObservation per variant, with experiment_run_id in its parameters), then flip it to running. An agent that creates an experiment does both in the same turn; if you create one via the API/CLI directly, do the same so it doesn’t sit silently recording nothing. Measurement needs each variant to be identifiable in the evidence data (e.g. one ad group per variant, or a landing-page path) so observations can be attributed. You don’t write a measurement recipe per experiment. A generic, config-driven recipe (recipe-experiment-measure) is the default: each variant declares how to filter its own rows per dataset ({ column, values }) and each metric declares the SQL that computes it (returning a value column, with {{variant_filter}} and {{window}} placeholders). The recipe loops metrics × variants and records the observations — no per-use-case code.

​
Observation types — measurements, forecasts, and orderings

The observation log holds more than metric reads. Three types matter when you’re reading whether the loop is working:
  • metric_read — what reality measured for a variant (cost per lead, conversion rate). This is the ground truth every other type is scored against.
  • prediction — a forecast a judge made before reality spoke: “I expect variant B to convert better.” A judge is any cheaper-than-reality opinion — a critic lens, a persona panel, a pairwise tournament, or a human’s answer to a choice. A prediction is only worth something once a later metric_read on the same variant lets you check it, so predictions are stored so they can be paired with the measurement that eventually arrives.
  • comparison — an ordering between variants rather than a value: “B beat control.” Two predictions imply an ordering, an explicit head-to-head records one directly, and a human choice tied to a running experiment writes one every time you pick — your pick marks the chosen variant as beating the ones you didn’t.
You read them the same way as any observation — erdo experiment observations <slug> --type prediction (or comparison) — and they carry the same append-only, RBAC’d guarantees as metric reads.

​
Judges and the calibration record

Predictions and comparisons are only useful if you find out how often they were right. That’s what calibration is: pairing each judge’s forecasts and orderings against the outcomes reality later measured, so a judge earns trust from its track record instead of assertion. The headline signal is pairwise agreement — of the variant pairs where both the judge and reality expressed an ordering, how often the judge ordered them the way reality did. It covers prediction rows (two forecasts imply an ordering) and comparison rows (an explicit ordering) uniformly, against reality’s ordering from the measured metric reads. Alongside it you get raw prediction and comparison counts, how many predictions reality has caught up with (paired predictions), and — because a human choice is just another judge — the same reading for your own answers. The calibration readout is the trust dial: it’s a query over the ledger, not a separate subsystem, so it stays honest as more observations land. A landing-page A/B test is the fullest example: the automatic critics predict which page will convert, your attention-feed choices record human orderings, and the lead dataset supplies the metric_read that scores both — so over time you learn which judge to trust before you’ve spent the traffic to be sure. Default judges are platform-calibrated, so they aren’t blind on day one. A new workspace has no paired outcomes for weeks, which would leave the default lenses Erdo ships with no track record. For those default lenses only, the calibration prior pools statistics across every workspace — agreement rates and sample counts, never anyone’s page content, verdicts, or identity — so a default judge starts from the whole platform’s record. That prior is then adjusted by your own paired outcomes as they accumulate: with enough of your own results a judge whose reality diverges from the platform’s overrides the shared prior. Judges you author stay private to your workspace and never pool. While a judge’s local record is still thin, Erdo keeps a slightly larger share of bets on the random arm, so your own traffic confirms or refutes the shared prior faster.

​
The judge profile page

Every judge has a profile — open one from the Judges tab under Activity, or from an item’s actor chip. A judge is a review lens (a critic skill): its profile shows the rubric it grades against, the artifact kind it screens, and whether it’s a default lens Erdo ships or one generated for your org. Default lenses carry bootstrap trust and may block a build before any calibration exists; a generated judge enters untrusted and has to earn screening power from its record. Below that is the judge’s calibration, made legible without a query: its pairwise agreement and sample counts, a breakdown by artifact kind, and how many of its scored pairs came from the random arm — the bets it did not gate, the only place its screening value is directly measurable. The misordered pairs panel is the point: each row is a case where reality ranked two variants the opposite way the judge did — grounded numbers, not prose. This is what answers “why did the critic block this?” without touching SQL, and it’s also the seed the engine uses to generate a sharper judge. A recent predictions list rounds it out with the forecasts the judge has been making lately across every experiment you can see.

​
The decision policy

For page experiments, prose decision rules and fixed sample sizes are slow — our traffic takes weeks to reach a fixed N, and nothing moves traffic while the evidence accumulates. The decision policy replaces both. It runs as the measurement recipe’s final step and does three things from the same per-variant counts the recipe already read:
  • Belief. Each variant gets a Beta-Binomial posterior on the primary metric’s rate. If the persona panel has a calibrated prediction for the variant, that prediction seeds the posterior as pseudo-sessions — but only once the calibration ledger holds enough paired experiments and its falsification test passes; until then the prediction carries zero weight and the belief is data-only. A calibrated prediction is capped so a real read of a few hundred sessions always dominates it.
  • Allocation. A constrained Thompson-sampling step nudges each variant’s allocation_percent toward its posterior win-frequency, but never more than 15 points a day, never below a 10% floor while a variant is live, and never dropping the control below 20% until the experiment decides. The caps make it boring by construction: a bad prior or a bad day shifts spend modestly, never zeroes a variant out. A step that moves more than 30 points in total raises an attention item.
  • Stopping. A posterior rule replaces the prose one: ship a variant when it’s ≥ 95% likely to beat the control (with an expected-loss guard against winning by a hair on huge uncertainty), kill it when it’s ≤ 5% likely, or declare inconclusive at a 28-day horizon. A guardrail whose posterior is > 90% likely to have regressed vetoes a ship. When the rule settles on ship or stop, it fires the same decision loop the prose path uses.
The written outcome also reports a prediction-augmented effect (an HAIPW estimate) alongside the naive one — a tighter confidence interval when the prediction is good, and never a biased point estimate when it isn’t. It’s reported, not gating: decisions ride the posterior rule above. Read the policy’s latest state — per-variant posteriors, current allocation, win frequencies, stop probabilities, the calibration gate, and the recommended decision — with erdo experiment policy <slug>, the erdo_get_experiment_policy MCP tool, or GET /v1/experiments/:slug/policy. It’s also attached to the experiment GET response as policy_state.

​
Programmatic access

Everything below is org-scoped and RBAC’d (project view for reads, contribute for writes) and referenced by slug.

​
CLI

Each --variant is comma-separated key=value pairs plus the bare flag control (marks the baseline). page=<artifact-id> is shorthand for type=page + that resource id — it links the variant to the page it serves. alloc= is the traffic split (0–100). Each --evidence is slug=<dataset>,role=primary or the shorthand <dataset-slug>:<role> (role defaults to primary). tournament requires --criteria and accepts optional --kind, repeated --variant <key> flags to restrict the field, and --max-comparisons; it runs synchronously and can take minutes. record-decision logs an allocation override — the variant, the allocator’s --recommended action, the action you --chosen, and a --reason — and that disagreement becomes calibration data on the allocator itself.

​
MCP tools

​
REST

Base URL https://api.erdo.ai. Authorization: Bearer <token> + X-Organization-ID. /v1/experiments/:slug/tournament requires criteria in the body and accepts optional kind, variant_keys, and max_comparisons; it responds synchronously with the ranking (can take minutes). /v1/experiments/:slug/allocation-decisions takes variant_key, recommended_action, chosen_action, and reason. /v1/experiments/:slug/observations takes optional variant_key, observation_type, judge_slug (filter to one judge’s forecasts within the experiment), and limit query parameters. /v1/judge-calibration takes optional project_slug (limit to one project; omit to span every project you can view) and limit (max ledger rows scanned) query parameters.