> ## Documentation Index > Fetch the complete documentation index at: https://docs.erdo.ai/llms.txt > Use this file to discover all available pages before exploring further. # Judges and calibration > A judge is anything that predicts which variant wins before reality measures it — a critic lens, a persona panel, a pairwise tournament, your own answer to a choice. Calibration is the track record that says which ones to trust. # Judges and calibration Real traffic is slow and expensive: it takes weeks and hundreds of dollars to find out which landing-page variant actually converts. A **judge** is anything cheaper than reality that predicts that answer *before* reality speaks — a critic lens grading a page, a [persona panel](/persona-panel) forecasting conversions, a [pairwise tournament](#pairwise-tournaments) ranking a field, or your own answer to a [choice](/attention) in the Activity feed. Judges are fast, synthetic, and fallible, so none of them settles a bet on its own. What makes them useful is that every prediction is checkable: when reality later measures the same variant, you find out whether the judge was right. That check is the whole discipline. A judge with a good track record earns the power to screen out weak variants before you spend on them; a judge whose predictions stop matching reality loses that power automatically. The rest of this page is how that track record is kept, and the surfaces for running the judges and reading their records. ## Calibration — the trust dial **Calibration** is the pairing of a judge's forecasts against the outcomes reality later measured. Because a judge's predictions and reality's measurements land in the same [experiment observation ledger](/experiments) — with the same variant and metric keys — calibration is a **query over that ledger, not a separate subsystem.** It stays honest as more observations land, because there is nothing to keep in sync: the same rows that record what happened also record what was predicted. The headline number is **pairwise agreement**: of the variant pairs where both the judge and reality expressed an ordering, how often did the judge order them the way reality did. It covers `prediction` rows (two forecasts imply an ordering) and `comparison` rows (an explicit head-to-head) uniformly, each side oriented by its own metric's direction, so a judge that picked the *lower*-cost-per-lead variant scores as agreeing rather than disagreeing. Alongside agreement you get the raw prediction and comparison counts, how many predictions reality has caught up with (*paired predictions*), and a breakdown **by artifact kind** — because a judge validated on landing pages tells you nothing about how it grades dashboards, and its screening power is scoped to the kind it has actually earned it in. Read the calibration record on any surface: ```bash theme={null} # every judge's paired predictions and pairwise agreement vs measured reality erdo experiment calibration --project acme # one judge's track record erdo experiment calibration --judge artifact-critic ``` The MCP tool is `erdo_judge_calibration`; the REST mirror is `GET /v1/judge-calibration` (optional `project_slug` to scope to one project, `limit` to cap ledger rows scanned). Each judge also has a profile page under **Activity → Judges** that renders the same record without a query — its rubric, the kind it screens, whether it's a default lens Erdo ships or one generated for your org, and a **misordered pairs** panel showing exactly the cases where reality ranked two variants the opposite way the judge did. **Default judges are platform-calibrated, so they aren't blind on day one.** A new workspace has no paired outcomes for weeks. For the default lenses Erdo ships only, the calibration prior pools *statistics* across every workspace — agreement rates and sample counts, never anyone's page content, verdicts, or identity — so a default judge starts from the whole platform's record and is then pulled toward your own outcomes as they accumulate. Judges you author stay private to your workspace and never pool. ## Pairwise tournaments When a fan-out produces many variants of one brief, ranking them is the first rung of judgment. Erdo never asks a judge for an absolute score, because LLM judges are markedly more reliable at *"which of these two is better"* than at *"rate this one out of ten."* So a **pairwise tournament** compares variants two at a time and fits the verdicts into a ranking. The ranking is a **Bradley–Terry** fit, not a knockout bracket, and the reason is that pairwise preferences are legitimately intransitive — a panel can produce A beats B, B beats C, and C beats A. A bracket would turn that cycle into an arbitrary winner decided by the seeding. Bradley–Terry instead fits one latent strength per variant that best explains *all* the comparisons at once, so a variant that drops one comparison but wins many others still ranks above one that won that comparison and little else. Presentation order is randomized per pair (and recorded), which turns an LLM's position bias into noise the fit averages out; and near-duplicate variants are collapsed first so the tournament doesn't waste comparisons distinguishing a page from its own regeneration. Every comparison the tournament makes persists to the experiment ledger as a `comparison` observation, so the pairwise judge earns a calibration track record exactly like every other judge. Run one: ```bash theme={null} erdo experiment tournament acme-cpl-ab --criteria "which page will convert cold paid traffic best" \ --kind landing_page --max-comparisons 20 # restrict the field with repeated --variant flags erdo experiment tournament acme-cpl-ab --criteria "clarity of the offer" --variant b --variant c ``` The MCP tool is `erdo_run_pairwise_tournament`; the REST mirror is `POST /v1/experiments/:slug/tournament` (body: `criteria` required; optional `kind`, `variant_keys`, `max_comparisons`). The call is **synchronous and can take minutes** — it runs the comparisons before it returns the ranking, the merges it made, and what it persisted. ## Judge back-test — standing from the first minute A newly created judge has no track record, so it would sit advisory for weeks — unable to earn screening power from the very reality it was created to grade — because predictions only persist for variants that happen to be critiqued while an experiment is live. The **judge back-test** closes that gap: it runs the judge over your organization's currently-live experiments' bound variant pages *of its artifact kind* and persists one prediction per variant, so the judge earns (or fails to earn) calibration against outcomes you already know within its first minutes. The predictions are ordinary ledger rows, so they pair against the measured outcomes with no special-casing. It runs automatically when a judge is created **and whenever a judge's content changes** — an edited rubric, or a revision to the shared principles it reviews against. Re-runs are **idempotent per judge version**: a variant this judge has already forecast at its current content is skipped, while a genuinely changed judge re-forecasts each live variant exactly once under its new standard. When a *trusted* judge's re-screen finds blocking issues on variants serving live traffic, it raises a [judge re-screen item](/attention) in the attention feed proposing an iterate bet — it never rewrites a live page itself. ```bash theme={null} erdo judge backtest landing-conversion-v2 ``` The MCP tool is `erdo_backtest_judge`; the REST mirror is `POST /v1/judge-backtests` (body: `judge_slug`). It emits a summary of the predictions written. ## You are the most expensive judge Your own judgment is the most trusted judge in the cascade, so Erdo spends it sparingly and records it carefully. When the Activity feed asks you to [pick between variants](/attention) — a choice item tied to a running experiment — your answer is recorded as a `comparison`: the option you picked is marked as beating the ones you didn't, attributed to `human:`. That verdict does two things at once. It steers the live decision, and it **scores the cheaper judges against you** — every automatic judge that predicted the pair the way you did gains agreement, and every one that didn't loses it. So the minutes you spend answering choices aren't just deciding one experiment; they're sharpening the judges that will decide the next hundred without you. Because a human choice is just another judge, the calibration readout reports your own agreement rate the same way it reports a critic's — which is how you find out, over time, which automatic judge you can trust enough to stop being asked.