Judges and calibration
Real traffic is slow and expensive: it takes weeks and hundreds of dollars to find out which landing-page variant actually converts. A judge is anything cheaper than reality that predicts that answer before reality speaks — a critic lens grading a page, a persona panel forecasting conversions, a pairwise tournament ranking a field, or your own answer to a choice in the Activity feed. Judges are fast, synthetic, and fallible, so none of them settles a bet on its own. What makes them useful is that every prediction is checkable: when reality later measures the same variant, you find out whether the judge was right. That check is the whole discipline. A judge with a good track record earns the power to screen out weak variants before you spend on them; a judge whose predictions stop matching reality loses that power automatically. The rest of this page is how that track record is kept, and the surfaces for running the judges and reading their records.Calibration — the trust dial
Calibration is the pairing of a judge’s forecasts against the outcomes reality later measured. Because a judge’s predictions and reality’s measurements land in the same experiment observation ledger — with the same variant and metric keys — calibration is a query over that ledger, not a separate subsystem. It stays honest as more observations land, because there is nothing to keep in sync: the same rows that record what happened also record what was predicted. The headline number is pairwise agreement: of the variant pairs where both the judge and reality expressed an ordering, how often did the judge order them the way reality did. It coversprediction
rows (two forecasts imply an ordering) and comparison rows (an explicit head-to-head) uniformly,
each side oriented by its own metric’s direction, so a judge that picked the lower-cost-per-lead
variant scores as agreeing rather than disagreeing. Alongside agreement you get the raw prediction
and comparison counts, how many predictions reality has caught up with (paired predictions), and a
breakdown by artifact kind — because a judge validated on landing pages tells you nothing about
how it grades dashboards, and its screening power is scoped to the kind it has actually earned it in.
Read the calibration record on any surface:
erdo_judge_calibration; the REST mirror is GET /v1/judge-calibration (optional
project_slug to scope to one project, limit to cap ledger rows scanned). Each judge also has a
profile page under Activity → Judges that renders the same record without a query — its rubric,
the kind it screens, whether it’s a default lens Erdo ships or one generated for your org, and a
misordered pairs panel showing exactly the cases where reality ranked two variants the opposite
way the judge did.
Default judges are platform-calibrated, so they aren’t blind on day one. A new workspace has no
paired outcomes for weeks. For the default lenses Erdo ships only, the calibration prior pools
statistics across every workspace — agreement rates and sample counts, never anyone’s page content,
verdicts, or identity — so a default judge starts from the whole platform’s record and is then pulled
toward your own outcomes as they accumulate. Judges you author stay private to your workspace and
never pool.
Pairwise tournaments
When a fan-out produces many variants of one brief, ranking them is the first rung of judgment. Erdo never asks a judge for an absolute score, because LLM judges are markedly more reliable at “which of these two is better” than at “rate this one out of ten.” So a pairwise tournament compares variants two at a time and fits the verdicts into a ranking. The ranking is a Bradley–Terry fit, not a knockout bracket, and the reason is that pairwise preferences are legitimately intransitive — a panel can produce A beats B, B beats C, and C beats A. A bracket would turn that cycle into an arbitrary winner decided by the seeding. Bradley–Terry instead fits one latent strength per variant that best explains all the comparisons at once, so a variant that drops one comparison but wins many others still ranks above one that won that comparison and little else. Presentation order is randomized per pair (and recorded), which turns an LLM’s position bias into noise the fit averages out; and near-duplicate variants are collapsed first so the tournament doesn’t waste comparisons distinguishing a page from its own regeneration. Every comparison the tournament makes persists to the experiment ledger as acomparison
observation, so the pairwise judge earns a calibration track record exactly like every other judge.
Run one:
erdo_run_pairwise_tournament; the REST mirror is
POST /v1/experiments/:slug/tournament (body: criteria required; optional kind, variant_keys,
max_comparisons). The call is synchronous and can take minutes — it runs the comparisons before
it returns the ranking, the merges it made, and what it persisted.
Judge back-test — standing from the first minute
A newly created judge has no track record, so it would sit advisory for weeks — unable to earn screening power from the very reality it was created to grade — because predictions only persist for variants that happen to be critiqued while an experiment is live. The judge back-test closes that gap: it runs the judge over your organization’s currently-live experiments’ bound variant pages of its artifact kind and persists one prediction per variant, so the judge earns (or fails to earn) calibration against outcomes you already know within its first minutes. The predictions are ordinary ledger rows, so they pair against the measured outcomes with no special-casing. It runs automatically when a judge is created and whenever a judge’s content changes — an edited rubric, or a revision to the shared principles it reviews against. Re-runs are idempotent per judge version: a variant this judge has already forecast at its current content is skipped, while a genuinely changed judge re-forecasts each live variant exactly once under its new standard. When a trusted judge’s re-screen finds blocking issues on variants serving live traffic, it raises a judge re-screen item in the attention feed proposing an iterate bet — it never rewrites a live page itself.erdo_backtest_judge; the REST mirror is POST /v1/judge-backtests (body:
judge_slug). It emits a summary of the predictions written.
When the lenses read a page
When Erdo builds a landing page it reviews its own work before handing it over: the lenses for that kind read the finished page with fresh context, and anything they consider blocking goes back to the builder to fix. A page you write some other way gets no such pass —erdo pages update, the /v1
API, the page tool over MCP, a script, an automation, or a rollback to an earlier version all change
what visitors see without a build ever happening.
A review is not free: it renders the page at desktop and mobile and spends a model call per lens. So
Erdo does not re-run them every time a page changes — a reworded headline or a colour tweak leaves a
page that the lenses already passed, and paying for a fresh verdict on it buys nothing. They run in
three cases.
A page’s first content. There is no earlier verdict to lean on, and a page whose first version is
broken is broken for every visitor it ever gets. Builds cover this themselves; a page deployed
through the API or the CLI is reviewed a few minutes after it lands.
A full-scale rewrite. When a write replaces the bulk of a page, the previous verdict was about a
page that no longer exists. “The bulk” is measured mechanically — what share of the page’s lines are
new since the lenses last looked, against a fixed threshold of half the page — so it is a question of
size, not an interpretation of what your change meant. Ordinary edits are nowhere near it: replacing
one whole section of a twelve-section page is under a tenth. And because the comparison is against
the last reviewed version rather than the last write, a page that drifts through many small edits
still earns a review once those edits add up.
When you ask. Whoever is making the change is the one who knows whether it was cosmetic or
load-bearing, so any writer can request a review on the spot instead of waiting for a rewrite big
enough to trigger one automatically. The agent sets request_review on update_html_artifact /
edit_html_artifact and gets findings back inline, fixing what it agrees with in the same turn.
Writing through erdo pages update, the /v1 REST mirror, or the erdo_update_page MCP tool, the
same request_review boolean (--request-review on the CLI) runs the identical lenses and returns
the findings under review in the response. That block also carries the raw mobile lighthouse
measurement when PageSpeed answers: performance, accessibility, best_practices, and seo, with
each category optional so a failed or null category is absent rather than shown as zero. This inline
result is evidence for the writer that asked; requesting it does not create a separate Activity item.
Use it for a material edit to a live page: a restructured section, compliance or legal wording,
anything touching the form, the CTA, or layout. Skip it for a cosmetic tweak; a colour or a word
isn’t worth the render and the model call.
The lenses that run on an existing page are the ones that catch defects a change can introduce
regardless of intent — an element colliding with another at some breakpoint, content stranded behind
the sticky bar, a form that no longer submits, copy that reads as build scaffolding rather than as
something addressed to a visitor. The lenses that judge taste and strategy deliberately sit these
out, because an edit you made on purpose is not a mistake for a reviewer to relitigate.
When a background review finds something blocking, it becomes one
judge re-screen item in the Activity feed, marked higher when the page is publicly
shared than when it is still a draft. Nothing is rewritten on your behalf and the page keeps serving
exactly what it served before — the item names the defect, quotes the lens, and leaves the fix to you
or to the agent. It closes itself once a later review of that page comes back clean.
A few lenses never gate anything because their subject is a judgment call that belongs to a person —
notably fair-housing language on real-estate pages, where the same phrase can be a routine amenity
description on one page and unlawful steering on another, and an automatic rewrite of legally-sensitive
copy is its own risk. Those lenses report instead: on a page a build or an agent is writing, their
findings ride back to that agent inline; on a page nobody is watching — written through the API, the
CLI, a script, or a rollback — their findings become a compliance review item in the Activity
feed naming the lens, quoting the phrase, and proposing an inclusive rewrite that keeps the selling
point. The framing is deliberately careful: the lens surfaces candidate language for you (or your
compliance reviewer) to judge — it is not a legal determination, nothing is changed automatically,
and the item closes itself once a later review’s advisory lenses come back clean. New pages get the
benefit earlier still: every landing-page build carries inclusive-housing copy guidance at write
time, so resident-profiling phrasing is less likely to reach the page at all.
The two automatic triggers need nothing enabled and nothing called — the review follows the page, not
the tool that wrote it. request_review is there for the times you don’t want to wait: a specific
change you want checked now, on whichever surface made it.
You are the most expensive judge
Your own judgment is the most trusted judge in the cascade, so Erdo spends it sparingly and records it carefully. When the Activity feed asks you to pick between variants — a choice item tied to a running experiment — your answer is recorded as acomparison: the option you picked is
marked as beating the ones you didn’t, attributed to human:<you>. That verdict does two things at
once. It steers the live decision, and it scores the cheaper judges against you — every automatic
judge that predicted the pair the way you did gains agreement, and every one that didn’t loses it. So
the minutes you spend answering choices aren’t just deciding one experiment; they’re sharpening the
judges that will decide the next hundred without you.
Because a human choice is just another judge, the calibration readout reports your own agreement rate
the same way it reports a critic’s — which is how you find out, over time, which automatic judge you
can trust enough to stop being asked.