Skip to main content

​
The wind tunnel

Paid traffic is the most expensive judge you own. A statistically useful read on one landing-page variant costs hundreds of dollars of clicks, and a field of five variants costs five times that — most of it spent measuring the losers. The wind tunnel is what you run first: a calibrated panel of synthetic customers reads each variant, predicts how it will perform, and kills the obvious losers for cents rather than hundreds of dollars, so real traffic only ever has to decide among variants that already look plausible. The name is the honest framing. Aircraft designers don’t trust a wind tunnel because it is the sky — they trust it because they have measured, across many real flights, exactly how tunnel readings map to flight behaviour. Erdo’s works the same way: every prediction it makes is later paired against what real visitors actually did on the same variant, so the tunnel earns its trust from a track record you can check — and it tells you when it doesn’t know. The wind tunnel prunes; it never concludes. Real traffic is the only judge Erdo lets settle a bet.

​
Two stages: rank first, then diagnose

The wind tunnel runs in two stages, and they answer two different questions. Stage A — the persona panel answers which variants win. A jury of synthetic visitors, each built from one of your brand’s buyer cohorts, looks at phone screenshots of every variant and predicts, as per-visit probabilities, the same micro-conversions your live pages emit — scrolling to the end, clicking the call-to-action, starting the form, submitting a lead. Each persona reads each variant several times and the forecasts are averaged, because single reads of a synthetic visitor are noisy in a way repeated draws cancel. The panel ranks the field and flags a kill list — the variants predicted at less than half the leader’s rate. The full mechanics, and how personas are built from your brand brief and re-grounded in your real traffic, live on the persona panel page. Those numbers are uncalibrated forecasts until reality pairs with them. Models are good at ranking variants and bad at magnitudes, so the ordering is the signal you act on early; the absolute probabilities only start to mean something once enough real measurements have been paired against them. That pairing is the calibration record — the trust dial that decides how much weight a panel’s predictions carry. Stage B — diagnostic sessions answer why the losers lose. For the top survivors, each persona actually browses the page in a real mobile browser session and narrates its walk: what it read, where its attention broke, where it gave up and why. Stage B is deliberately diagnostic only — a handful of browsing agents is far too small a sample to estimate conversion, so it never produces a number, only the story behind the panel’s numbers. The browsing personas are even barred from submitting forms, because the surviving pages are live and a submission would write a fake lead into a real dataset. Diagnosis explains why a page loses; it is structurally prevented from pretending to measure how much.

​
How it composes with the rest of the loop

The wind tunnel is one rung of a larger ladder, and each rung is cheaper than the one below it:
  1. Fan out. An agent generates many independent variants of one brief — different personas, different angles — each built blind to its siblings so the diversity is real.
  2. Coalesce and prune. Near-duplicate variants are collapsed, then a pairwise tournament ranks the survivors, and the persona panel prunes the field by predicted performance. Only variants that clear the bar get real exposure.
  3. Let traffic decide. Survivors enter a running experiment. The public page resolver splits traffic across them, and the assigned variant is stamped into every lead row and analytics event, so the outcome data is variant-joined by construction — never by trusting a transform to copy a field.
  4. Calibrate. Because the panel’s predictions and reality’s measurements land in the same observation ledger with the same variant and metric keys, every real outcome becomes the pair for a prediction that came before it. The judges that predicted well earn screening weight; the ones that didn’t lose it. The tunnel gets more accurate on your pages as your own traffic accumulates.
This is what separates the wind tunnel from a one-off scoring tool: it is inside the loop that grades it. Evals keep Erdo honest about what it built (does the page render, does the form persist a lead); the wind tunnel keeps it honest about what it predicted (will the page perform). A page can pass every eval and still be a variant the panel correctly kills.

​
Honest limits

The best published systems anywhere for predicting A/B winners are right roughly three times out of four — a one-in-four wrong-winner rate. That is why the panel is a prior, not a verdict: calibration transports only to the kinds of change it was fitted on, so Erdo keeps a permanent share of variants going straight to real traffic past the judges, and lets every predictive layer lose influence automatically the moment its track record stops earning it. The wind tunnel’s job is to make sure you rarely pay to learn something a few cents could have told you first — not to replace the judge that ultimately counts.

​
Running the wind tunnel

The wind tunnel is part of how Erdo operates experiments, not just an endpoint — the agent that runs your workstreams carries the panel and the diagnosis sessions as its own tools and runs them on any new variant set before paid traffic is spent. So when experiments run autonomously, the wind tunnel is already in the loop. You can also drive each stage directly. Ask an agent in a thread:
“Run the persona panel on the hero-copy experiment and kill the losers, then diagnose the top three.”
Or run the stages yourself — full flags and output on the persona panel page:
The MCP tools are erdo_run_persona_panel and erdo_diagnose_experiment_personas; the REST mirrors are POST /v1/experiments/:slug/panel and POST /v1/experiments/:slug/diagnose. Both stages write ordinary experiment observations — the panel’s forecasts as prediction rows and its ranking as a decision_check, the diagnosis walks as action_taken rows — so you read their output the same way you read any other evidence, and calibration happens automatically as reality catches up.