update_dataset_schema through find_tools. It should first consult
list_dataset_purposes to reuse the current write target.
The agent supports column additions and schema declarations. Removing, renaming,
or retyping stored columns requires the operator CLI or API.
A list built from research can say where each row’s facts came from. Give its
provenance columns a semantic_role: source_url on the source page or
filing, source_quote on the exact supporting words, and source_seen_at on
when the source was read. Several columns can carry the same role. A companies
list might have website, employee_count, employee_count_url,
employee_count_excerpt and checked_at, with the three roles on the last
three. Those columns then show as the row’s sources, not as facts about the
company. Set the role even on a column named source_url: without one, a
column of that name is read as how a lead arrived. To make every write carry a
source, also turn on provenance_required
for the same columns (see Dataset ownership).
Read the current contract
Check what the dataset already declares before you change it. From the CLI, runerdo datasets schema <slug> --declared. Over REST, send
{"view": "declared_schema"} to POST /v1/datasets/:datasetSlug/fetch; the
MCP tool erdo_fetch_dataset_contents takes the same view argument. The
answer carries the contract under declared_schema, reads no rows, and leaves
declared_schema out when the dataset declares no contract. An API key scoped
to datasets:query for the dataset can make this call.
For a dataset with permanent lead identity, the read contract also includes
erdo_conversion_id, an internal text projection of the capture conversion
identity. Do not include this read-only column in a producer’s declaration or
write it as a form answer.
Add a column
Useadd_column to extend an existing dataset. This preserves its rows,
existing column definitions, and permanent lead identities. Do not create a
second leads dataset to add a field.
add-broker.json, then run:
POST /v1/datasets/acme.leads/schema. In MCP,
use erdo_update_dataset_schema with dataset_slug and the operations.
A populated dataset using permanent lead identity normally accepts only additive
column operations. Send additions separately from declare_schema, class changes,
removals, renames, or type changes. The platform-owned canonical_lead_id
column cannot be changed. Existing records have no answer for a newly added
question; the addition does not invent historical answers.
Add a money column
Declare itscurrency as a three-uppercase-letter ISO 4217 code and write
integer minor units. For example, USD 12.50 is the value 1250:
declare_schema, put currency beside the column’s
name and type. The type may be omitted; text and float types are rejected
when declaring currency. erdo datasets schema <slug> --declared shows the
currency alongside the column’s other metadata.
Budget answers from a lead form
A landing page’s budget question is the lead form’s budget field, and you do not map it in the pipeline. Erdo writes the answer onto five columns of the leads dataset and declares each with the role of the same name:
Read a lead’s budget by these roles rather than by column name. Leads captured
before a page used the budget field keep their answer in whatever column that
page wrote (
budget, budget_range and the like); those rows are not rewritten.
The columns are declared on the first capture that carries a budget, including
on a populated dataset with permanent lead identity. To declare one ahead of
that, add it with its role. A role is accepted only on a new, empty column of
the same name; a column that already holds values cannot gain one:
Declare a legacy lead dataset without rewriting it
An older lead dataset may already contain permanent identities but have no declared schema. Its existingdeclare_schema operation can establish the
contract without rewriting any stored rows. Send a single declaration that
explicitly includes every stored column and the platform identity definition:
SELECT * FROM data LIMIT 0
show the current queryable header without returning lead values. Exclude the
read-only erdo_conversion_id projection from the declaration; the write owner
checks the actual stored header again under its lock. A column absent from a
cached analysis is not evidence that it is new or safe to retype.
All other columns must remain optional. Existing business columns may omit
type, which preserves their stored type hints and inferred read types. An
explicit type must exactly match an existing stored hint; a CSV header alone
cannot prove a type. New optional producer columns may declare their types.
The platform checks the latest declaration, identity enrollment and file header
under the same write lock used by capture. It refuses missing physical columns,
new required business fields, identity changes, unproven retypes, mixed file
operations and class changes. This exception applies only while the declaration
is absent; it does not replace an existing contract or grant permission to
rewrite enrolled lead rows.
Change a purpose and its schema together
A combined purpose change and declaration authorizes the existing purpose carriers first. Its contract, class, purpose and registry metadata commit together; a refused move or failed final metadata write leaves the previous purpose and contract in place. Stored file rewrites and type-hint changes cannot roll back with that metadata. A request that supplies a purpose (including its current value) or clears it cannot also physically edit an existing file; it is refused before the file changes. First set the purpose with its complete metadata-only declaration; then send physical column edits separately without a purpose field. Unwritten datasets and metadata-only declarations continue through their existing schema checks.Connect and verify the form
After adding the column, update the transform to include the submitted field and normalize localized answers to a consistent value. Test the transform with representative payloads, save the pipeline, and verify a marked test submission in the dataset. Repeat for every page’s capture pipeline. Keep different questions in different columns. For example, “I am a broker” and “I am represented by a broker” are separate answers even if their labels share the word “broker.” A field required by the page should not make the pipeline reject partial lead submissions: keep the capture request’s required fields compatible with the form’s first submission.Synchronized row versions
Queries and CSV exports from synchronized datasets read the latest stored version of each provider row. Refreshing an existing row does not add another copy to ordinary reporting totals while warehouse compaction is pending. A sync can still publish different resources at different times; check its readiness and coverage before treating a report as complete. Empty synchronized tables keep their schema and a zero row count. Column statistics without observations, such as a minimum, percentile or average, remain absent; they do not remove the table from its technical analysis. Synchronized column statistics use consistent field names across database providers:dtype, unique_count, null_count and null_percent. Numeric
ranges use min and max; date and timestamp ranges use min_date and
max_date. Nullable categorical values retain their actual text. Unavailable
statistics remain absent instead of making the whole column catalog unreadable.
The column list remains complete when detailed statistics are skipped or fail.
Sampled statistics omit absolute distinct and null counts and whole-table null
percentages. SQL and Fabric catalog row estimates are also insufficient to
calculate a null percentage; a full-scan null count can still be reported
without one.
Sampled analyses identify their measurement scope in the summary as well as
the metadata. Adaptive SQL sampling reports the percentage actually queried.
When null statistics were skipped, data quality totals remain absent.
Fabric aggregate statistics use full-table queries; skipping expensive
percentiles does not make the resulting distinct or null counts sampled.
