Skip to main content

​
Judges and calibration

Real traffic is slow and expensive: it takes weeks and hundreds of dollars to find out which landing-page variant actually converts. A judge is anything cheaper than reality that predicts that answer before reality speaks — a critic lens grading a page, a persona panel forecasting conversions, a pairwise tournament ranking a field, or your own answer to a choice in the Activity feed. Judges are fast, synthetic, and fallible, so none of them settles a bet on its own. What makes them useful is that every prediction is checkable: when reality later measures the same variant, you find out whether the judge was right. That check is the whole discipline. A judge with a good track record earns the power to screen out weak variants before you spend on them; a judge whose predictions stop matching reality loses that power automatically. The rest of this page is how that track record is kept, and the surfaces for running the judges and reading their records.

​
Calibration — the trust dial

Calibration is the pairing of a judge’s forecasts against the outcomes reality later measured. Because a judge’s predictions and reality’s measurements land in the same experiment observation ledger — with the same variant and metric keys — calibration is a query over that ledger, not a separate subsystem. It stays honest as more observations land, because there is nothing to keep in sync: the same rows that record what happened also record what was predicted. The headline number is pairwise agreement: of the variant pairs where both the judge and reality expressed an ordering, how often did the judge order them the way reality did. It covers prediction rows (two forecasts imply an ordering) and comparison rows (an explicit head-to-head) uniformly, each side oriented by its own metric’s direction, so a judge that picked the lower-cost-per-lead variant scores as agreeing rather than disagreeing. Alongside agreement you get the raw prediction and comparison counts, how many predictions reality has caught up with (paired predictions), and a breakdown by artifact kind — because a judge validated on landing pages tells you nothing about how it grades dashboards, and its screening power is scoped to the kind it has actually earned it in. Read the calibration record on any surface:
The MCP tool is erdo_judge_calibration; the REST mirror is GET /v1/judge-calibration (optional project_slug to scope to one project, limit to cap ledger rows scanned). Each judge also has a profile page under Activity → Judges that renders the same record without a query — its rubric, the kind it screens, whether it’s a default lens Erdo ships or one generated for your org, and a misordered pairs panel showing exactly the cases where reality ranked two variants the opposite way the judge did. Default judges are platform-calibrated, so they aren’t blind on day one. A new workspace has no paired outcomes for weeks. For the default lenses Erdo ships only, the calibration prior pools statistics across every workspace — agreement rates and sample counts, never anyone’s page content, verdicts, or identity — so a default judge starts from the whole platform’s record and is then pulled toward your own outcomes as they accumulate. Judges you author stay private to your workspace and never pool.

​
Pairwise tournaments

When a fan-out produces many variants of one brief, ranking them is the first rung of judgment. Erdo never asks a judge for an absolute score, because LLM judges are markedly more reliable at “which of these two is better” than at “rate this one out of ten.” So a pairwise tournament compares variants two at a time and fits the verdicts into a ranking. The ranking is a Bradley–Terry fit, not a knockout bracket, and the reason is that pairwise preferences are legitimately intransitive — a panel can produce A beats B, B beats C, and C beats A. A bracket would turn that cycle into an arbitrary winner decided by the seeding. Bradley–Terry instead fits one latent strength per variant that best explains all the comparisons at once, so a variant that drops one comparison but wins many others still ranks above one that won that comparison and little else. Presentation order is randomized per pair (and recorded), which turns an LLM’s position bias into noise the fit averages out; and near-duplicate variants are collapsed first so the tournament doesn’t waste comparisons distinguishing a page from its own regeneration. Every comparison the tournament makes persists to the experiment ledger as a comparison observation, so the pairwise judge earns a calibration track record exactly like every other judge. Run one:
The MCP tool is erdo_run_pairwise_tournament; the REST mirror is POST /v1/experiments/:slug/tournament (body: criteria required; optional kind, variant_keys, max_comparisons). The call is synchronous and can take minutes — it runs the comparisons before it returns the ranking, the merges it made, and what it persisted.

​
Judge back-test — standing from the first minute

A newly created judge has no track record, so it would sit advisory for weeks — unable to earn screening power from the very reality it was created to grade — because predictions only persist for variants that happen to be critiqued while an experiment is live. The judge back-test closes that gap: it runs the judge over your organization’s currently-live experiments’ bound variant pages of its artifact kind and persists one prediction per variant, so the judge earns (or fails to earn) calibration against outcomes you already know within its first minutes. The predictions are ordinary ledger rows, so they pair against the measured outcomes with no special-casing. It runs automatically when a judge is created and whenever a judge’s content changes — an edited rubric, or a revision to the shared principles it reviews against. Re-runs are idempotent per judge version: a variant this judge has already forecast at its current content is skipped, while a genuinely changed judge re-forecasts each live variant exactly once under its new standard. When a trusted judge’s re-screen finds blocking issues on variants serving live traffic, it raises a judge re-screen item in the attention feed proposing an iterate bet — it never rewrites a live page itself.
The MCP tool is erdo_backtest_judge; the REST mirror is POST /v1/judge-backtests (body: judge_slug). It emits a summary of the predictions written.

​
You are the most expensive judge

Your own judgment is the most trusted judge in the cascade, so Erdo spends it sparingly and records it carefully. When the Activity feed asks you to pick between variants — a choice item tied to a running experiment — your answer is recorded as a comparison: the option you picked is marked as beating the ones you didn’t, attributed to human:<you>. That verdict does two things at once. It steers the live decision, and it scores the cheaper judges against you — every automatic judge that predicted the pair the way you did gains agreement, and every one that didn’t loses it. So the minutes you spend answering choices aren’t just deciding one experiment; they’re sharpening the judges that will decide the next hundred without you. Because a human choice is just another judge, the calibration readout reports your own agreement rate the same way it reports a critic’s — which is how you find out, over time, which automatic judge you can trust enough to stop being asked.