> ## Documentation Index > Fetch the complete documentation index at: https://docs.erdo.ai/llms.txt > Use this file to discover all available pages before exploring further. # JudgeCalibrationAPI mirrors erdo_judge_calibration. ## OpenAPI ````yaml /api/openapi.json get /v1/judge-calibration openapi: 3.0.0 info: title: Erdo API description: >- Erdo's REST API: query and write datasets, run agents, manage threads, integrations, pages, evals, workstreams, and experiments. Authenticate with a Bearer API key (erdo_api_...) or scoped token (erdo_token_...). version: '2026-07-25' servers: - url: https://api.erdo.ai description: Production security: - bearerAuth: [] paths: /v1/judge-calibration: get: summary: | JudgeCalibrationAPI mirrors erdo_judge_calibration. operationId: GET:mcp.JudgeCalibrationAPI parameters: - allowEmptyValue: true explode: true in: query name: project_slug required: false schema: type: string style: form - allowEmptyValue: true explode: true in: query name: limit required: false schema: format: int32 maximum: 2147483647 minimum: -2147483648 type: integer style: form responses: '200': content: application/json: schema: properties: defects: description: > ledger — the seeded complement to each judge's misordered\_examples. They are not attributed to any single judge (a defect scores no judge's ordering); it is a standalone failure signal judge generation reads to propose a lens that would have caught the broken page before it shipped. Bounded per read. items: $ref: '#/components/schemas/types.ExperimentDefect' title: > Defects are the human "this variant is broken" verdicts on the scanned type: array judged_bets: format: int64 type: integer judges: items: $ref: '#/components/schemas/types.JudgeCalibrationStats' type: array observations_scanned: description: > computation saw (after RBAC filtering) — a hint that the limit was hit. format: int64 title: > ObservationsScanned is how many prediction/comparison/metric_read rows the type: integer random_arm_bets: description: > judge screening (a variant carrying random\_arm=true) or was screened by at least one judge. The ratio is the permanent random arm's headline number: with too few random-arm bets the engine cannot know whether its judges beat random selection at all — the comparison needs exactly the labels the judges saved. Counted at the (experiment, variant) grain over measured variants. format: int64 title: > RandomArmBets / JudgedBets split every measured bet by whether it skipped type: integer type: object description: Success response default: $ref: '#/components/responses/APIError' components: schemas: types.ExperimentDefect: description: > — a 'defect' observation written when a human flagged a choice's variant. It is the seeded complement to the misordered-pair residual stream: a defect costs no conversion (a clipped hero, off-brand type, scaffolding text on a stat card), so it can never surface as a variant that "converted worse than predicted", yet it is exactly the kind of failure judge generation should learn to catch. JudgeSlug is "human:\"; Reason is the human's free text naming the problem. properties: experiment_run_id: type: string judge_slug: type: string observed_at: format: date-time type: string reason: type: string variant_key: type: string title: > ExperimentDefect is one human "this variant is broken" verdict from the ledger type: object types.JudgeCalibrationStats: description: > were later measured, and how often it ordered variants the way reality did. Pairwise agreement is the trust signal — it covers both prediction rows (two forecasts imply an ordering) and comparison rows (an explicit ordering) uniformly, against reality's ordering from the measured metric\_reads. properties: agreed_pairs: format: int64 type: integer by_artifact_kind: additionalProperties: $ref: '#/components/schemas/types.JudgeCalibrationKindStats' description: > details\_json->>'artifact\_kind' tag, when judges set it (e.g. landing\_page vs dashboard). Empty when no prediction carried a kind. title: | ByArtifactKind breaks the prediction-derived stats out by the type: object by_grounding_source: additionalProperties: $ref: '#/components/schemas/types.JudgeCalibrationKindStats' description: > prediction was made by an AUTHORED persona (brand-brief cohort) or a MEASURED one (grounded from real sessions) — keyed "authored"/"measured". This is the acceptance test for persona grounding v1: grounded personas must out-agree authored ones. Populated only for persona judges when the caller supplied the measured-hash set; empty otherwise. title: > ByGroundingSource splits this judge's pairwise agreement by whether the type: object comparable_pairs: description: > where BOTH the judge and reality expressed an ordering; AgreedPairs is how many the judge got right. PairwiseAgreementRate = AgreedPairs/ComparablePairs (0 when there are no comparable pairs yet). format: int64 title: > ComparablePairs is the number of variant pairs (within one experiment) type: integer direct_pairs: format: int64 type: integer effective_agreement: description: > platform prior adjusted by this org's local pairs as they accumulate. With zero local pairs it equals the platform rate; with many it converges to the org-local rate (a divergent org overrides the prior once it has the evidence). EffectivePairs is the combined sample size behind it (capped platform prior + local pairs) — the sample count the trust gate reads. See HierarchicalAgreement. title: > EffectiveAgreement is the hierarchical (Beta-binomial shrinkage) estimate: the type: number effective_pairs: format: int64 type: integer judge_slug: type: string misordered_examples: description: > of variant pairs where reality's measured ordering disagreed with the judge's (the surprises). These are the seed for judge GENERATION: a new judge is proposed to catch the failure these examples name. Capped per judge (judge generation wants the head of the stream, not its full history). items: $ref: '#/components/schemas/types.JudgeMisorderedPair' title: > MisorderedExamples is the residual stream for this judge — a bounded sample type: array paired_predictions: description: > metric\_read on the same (experiment\_run, variant\_key) — i.e. reality eventually spoke on that forecast. DirectPairs is the further subset where the measured metric\_name matched the forecast's (a value-vs-value pair, not only a ranking pair). format: int64 title: | PairedPredictions is the subset of predictions that have a later type: integer pairwise_agreement_rate: type: number platform_agreement: description: > These three fields are populated ONLY for app-level default judges (the NULL-org seeded lenses Erdo ships). An org-authored judge's record stays org-private and never pools, so they are zero for it. PlatformAgreement / PlatformPairs are the judge's agreement rate and paired count POOLED across every org — statistics only, never verdict content and never an org identifier (see the platform-calibration privacy contract). They let a default judge's trust cold-start from the whole platform's record while a new org has zero local pairs. title: | --- hierarchical (platform-pooled) calibration --- type: number platform_pairs: format: int64 type: integer random_arm_pairs: description: > against a variant that reached reality via the random arm (a bet flagged random\_arm=true, i.e. one that skipped judge screening and went to reality anyway — see the meta loop). The permanent random arm is how the engine measures what the judges cost: a judge's agreement over bets it did NOT gate is its track record on the counterfactual the screened path can never show. format: int64 title: > RandomArmPairs is how many of this judge's ComparablePairs were scored type: integer total_comparisons: format: int64 type: integer total_predictions: description: | judge across the scanned ledger. format: int64 title: > TotalPredictions / TotalComparisons are raw row counts attributed to this type: integer title: > JudgeCalibrationStats is one judge's track record: how often its predictions type: object types.JudgeCalibrationKindStats: description: > the "trust region" the input-dependent gate reads (Kennedy–O'Hagan): a judge is trusted to block a build only within an artifact kind its forecasts have been validated on. The Effective\* fields mirror the top-level hierarchical estimate but scoped to this ONE kind: kind-local pairs shrunk toward this kind's platform prior. A kind with no local or platform evidence has zero effective pairs, so the gate treats the judge as advisory there — exactly like an uncalibrated judge. properties: agreed_pairs: format: int64 type: integer comparable_pairs: format: int64 type: integer effective_agreement: type: number effective_pairs: format: int64 type: integer paired_predictions: format: int64 type: integer pairwise_agreement_rate: type: number platform_agreement: description: > (app-level default judges only; zero for org-authored judges, which never pool). EffectiveAgreement / EffectivePairs are the kind-scoped hierarchical estimate — the (rate, sample) the gate scores via TrustAgreementForKind. title: > PlatformAgreement / PlatformPairs are this kind's pooled prior across every org type: number platform_pairs: format: int64 type: integer total_predictions: format: int64 type: integer title: > JudgeCalibrationKindStats is the per-artifact-kind slice of a judge's record — type: object types.JudgeMisorderedPair: description: > preferred JudgeWinner over RealityWinner, but reality's measured Metric ranked RealityWinner higher. It is grounded — every field is a recomputable fact about what was measured, never a guess — so a judge proposed from it anchors to checkable reality, the anti-Goodhart property the meta loop depends on. properties: experiment_run_id: type: string judge_winner: type: string judge_winner_value: type: number metric: type: string reality_winner: type: string reality_winner_value: type: number title: > JudgeMisorderedPair is one residual: within one decided experiment the judge type: object responses: APIError: content: application/json: schema: externalDocs: url: https://pkg.go.dev/encore.dev/beta/errs#Error properties: code: description: Error code example: not_found externalDocs: url: https://pkg.go.dev/encore.dev/beta/errs#ErrCode type: string details: description: Error details type: object message: description: Error message type: string title: APIError type: object description: Error response securitySchemes: bearerAuth: type: http scheme: bearer description: An Erdo API key (erdo_api_...) or scoped token (erdo_token_...). ````