Dataset refreshes

A dataset refresh is the automation that keeps one dataset current. It belongs to the dataset itself: when you ask an agent to keep a dashboard fresh, Erdo configures that dataset’s refresh instead of creating a separate job that also writes to it. Acting platform monitors and experiment measurement refreshes remain active when nobody opens their output dataset. Their work is an alert or a measurement, so an unused output does not make the automation dormant. Ordinary refreshes still use recent views and manual activity when deciding whether to pause.

When refreshes run

On view is the default. A stale dataset refreshes when someone opens the page or dashboard that uses it, so it costs nothing while nobody is looking.
On a schedule is for data that must already be current before another consumer runs, such as a scheduled report.
On source writes runs a batch refresh after rows are committed to a named source dataset. It stays idle between writes.
Live refreshes use a verified webhook or supported polling source for genuinely streaming data.

Only one active execution runs for a dataset refresh at a time. Repeated page views or manual clicks join the run already in progress instead of starting a second writer. A refresh paused automatically for inactivity wakes on the first dashboard view and can refresh immediately, without waiting for another page load. Disabling the refresh automation stops automatic runs from page views, schedules, webhooks and automatic repair checks. A page’s Refresh button also respects that stop: forcing a freshness check does not enable a disabled automation. Enable the automation to restore those triggers. An authorized operator can still run it once from Automations or the dataset refresh command without enabling its schedule. A retired automation cannot run again. For table-class file datasets, a committed write also schedules an eventual catalog refresh. The refresh is provider-free: it rereads the stored file, updates the discovered tables and analysis, and coalesces duplicate delivery. If delivery is delayed or a refresh is interrupted, the existing periodic analysis pass is the fallback. It does not replay a paid integration lookup; file catalog repair reads only the dataset’s stored file. Periodic repair checks file catalogs in a separate fair queue. Each eligible written table is checked before repair work starts, so a failed or busy table waits for the retry interval instead of blocking newer files. The check record does not claim that analysis succeeded; the catalog is marked analyzed only after the existing storage checkpoint completes. Provider integrations have an independent bounded queue, so file failures do not consume their refresh window. Both queues retain daily analysis eligibility and a two-hour attempt cooldown; failed or canceled work stays stale and visible for repair. Metadata refresh also records admission before checking provider status. Skipped integrations rotate behind unchecked candidates with a thirty-minute attempt cooldown, while the normal provider-specific refresh interval still applies. Periodic integration analysis keeps its caller deadline through generation and persistence, so an interrupted pass cannot continue beyond its refresh window.

Script or agent

An agent can save through its scoped row-writing tool or through datasets.write in run_js tied to the same refresh. Either records a successful target write. It does not need to repeat a programmatic write just to record the outcome. A write to another dataset or an external service does not count. Erdo uses a deterministic script when the source and output shape are known. That costs one credit per run and uses no LLM. An agent-backed refresh is reserved for work that genuinely needs judgement or extraction on every run. Save account identifiers, rates and other recipe settings in the refresh’s parameters field rather than embedding them in source. Python and JavaScript recipes both read these values from PARAMETERS, including nested objects and arrays. Python also receives the runner’s dataset_id, dataset_slug, refresh_mode, key_column and context dataset_slugs when applicable. In Python, these names are reserved: recipe parameters cannot change the runner’s write target or policy, or supply a runner field that is absent. Use a separate name for recipe values. For example, with parameters set to {"settings":{"customer_id":"ACCOUNT_ID","rates":{"click":0.03}}}, a Python recipe can read settings = PARAMETERS["settings"]. The recipe must also limit which settings it includes in its emitted dataset. Deterministic JavaScript refresh recipes can:

read the existing target or another permitted dataset;
read public HTTP sources with GET or HEAD;
call connected-source actions that the integration explicitly declares as safe, read-only refresh sources; and
read saved recipe values from the uppercase PARAMETERS object.

They cannot call mutation-capable actions, make POST/PUT/DELETE requests, write another dataset directly, or call an LLM. The refresh executor owns the single target write, so approvals cannot accidentally turn a data reader into a second automation system. Experiment measurement recipes retain their dedicated typed observation and decision-signal primitives. Those platform-owned operations are not general provider actions or arbitrary dataset writes.

Bind an experiment measurement refresh

When a refresh measures an experiment, set experiment_run_id to that experiment’s UUID. This binds the refresh’s lifecycle to the experiment so its schedule is disarmed when the experiment settles. The experiment must be readable by the caller and belong to the same organization as the dataset. The CLI accepts --experiment-run-id <uuid>; the REST and MCP request field is experiment_run_id. Keep parameters.experiment_run_id too when your recipe uses it. If only that recipe parameter is supplied, it also establishes the binding. Invalid UUIDs and conflicting copies are refused before configuration. Read the structured binding back with erdo datasets refresh show <slug> or the refresh GET response’s experiment_run_id, independently of recipe parameters.

erdo datasets refresh set acme.measurements --strategy script_js \
  --transform-js @measure.js --parameters @measure.json \
  --experiment-run-id "$EXPERIMENT_ID" --schedule '0 7 * * *' --no-on-view

Trigger a refresh from source writes

Use source-write triggers for a queue or a derived dataset that should update when its input changes. Configure the existing refresh with event_dataset_slugs and, optionally, event_debounce_seconds. CLI equivalents are --event-datasets and --event-debounce:

erdo datasets refresh set acme.processed-requests --strategy script_js \
  --transform-js @process.js --mode upsert --key-column request_id \
  --event-datasets acme.requests --event-debounce 10 --no-on-view

Each source must be readable by the configuring user and belong to the target dataset’s organization. Up to 16 source references are allowed; references that resolve to the same dataset create one trigger. The debounce is an integer from 0 to 3600 seconds: it waits after the first write to combine a burst of writes, not a recurring polling interval. Values from 0 through 5 dispatch directly through the normal job admission path after that bounded wait. Longer values use the minute scheduler, which can add roughly one minute after the debounce. The scheduler also recovers short triggers left armed by cancellation or shutdown. Duplicate event deliveries do not postpone a pending run. Delivery is at least once, so a late redelivery may start another check; actions with side effects must retain their durable idempotence. Execution admission and capacity limits still apply. Source writes, a cron schedule and on-view refreshes can coexist. Source-triggered refreshes stay active during inactivity so a later write can wake them. Disabling or disarming the owner stops automatic source triggers too. These bindings apply to batch strategies, including scripts and agent runs; live webhooks do not use them. They do not grant a recipe additional data access. The refresh read response includes event_triggers, with each source’s dataset_id, debounce_seconds and effective enabled state. The configuration response returns the resolved event_dataset_ids. A full replacement must send the source bindings again; omitting them removes them. Pending work on an unchanged source transfers to the replacement owner. A new binding reacts to future writes; it does not replay source history. Run the refresh once to process an existing backlog. Deterministic recipes are also tested during configuration. If a recipe listens to writes to its own output, it must stop returning rows when its queue is drained. Returning an empty array leaves the dataset untouched and emits no write event. Returning the same non-empty rows can keep scheduling work, so bound retries and return only work that still needs processing.

Replace, upsert, and no changes

Replace applies a complete non-empty result and replaces the stored rows.
Upsert merges a non-empty result using the configured key column or composite key.
Returning an empty array means no changes in either mode. It preserves the existing dataset; it never clears it.

Replacing a file publishes the same dataset-rows-written event as other committed writes, after the storage swap succeeds. Because a replacement is a complete snapshot, its rows_affected value is -1 (unknown), not the number of rows in the snapshot. Event-triggered jobs, dataset catalogs and websocket readers still react to the committed change; consumers that specifically count new leads ignore nonpositive values. Use upsert for rolling windows, paged sources, and historical dashboards. Use replace only when every successful run returns the complete desired dataset. When you set the mode yourself this is the choice that costs the most to get wrong, because both modes succeed either way. replace deletes every row the recipe did not return on that run — including rows a dashboard was built on, and rows a source has simply stopped paginating back to. upsert merges on the key column and leaves everything outside the returned window alone, which is why anything with a rolling window or a pagination cursor wants it. An upsert without a key column is refused rather than guessed at. For an agent-backed upsert, a matching row is replaced by default. An agent can use a partial merge to fill blank fields while preserving existing non-empty values, and name the fields it intends to overwrite. This supports enrichment without discarding earlier evidence or contacts. A configured full replacement still requires the complete result and cannot accept partial merge options. An agent-backed run interrupted by a backend update can resume under the same execution. It reports success after confirming its target write or explicit no-change outcome and validating the result. A missing outcome or invalid result remains a failure in Activity. During an agent refresh, inline JavaScript datasets.write calls use the same owning dataset, configured mode and key as the refresh’s dataset-write tool. Only a committed write earns a refresh receipt; a script’s returned claim of a write does not. A write committed before a later script error remains recorded, while the error is still reported to the agent. A confirmed batch write does not establish that every record in a research backlog has been completed. A run that replaces the refresh instructions while it works still settles the dataset’s own refresh status. The confirmed write it already made is what that status is recorded from, so the dataset shows the run’s real outcome rather than staying on the previous run’s result with a refresh apparently still in flight. Every run records its outcome in Activity. Authenticated source calls are also recorded by app and action name, without storing their input parameters or returned source data.

Editing a refresh

Ask an agent to update the dataset refresh, or reconfigure it from the dataset. The generic automation editor can rename, enable, or disable the schedule, but it cannot replace refresh instructions or source code: those stay with the dataset configuration that owns the behavior. The same configuration is also available without a chat session — erdo datasets refresh set on the CLI, erdo_configure_dataset_refresh over MCP, and PUT /v1/datasets/:datasetSlug/refresh — which is what you want when a refresh is part of something you are scripting, or you are giving several datasets the same treatment. Editing an existing refresh is that same call: it replaces the dataset’s refresh whole and retires the schedule it superseded, so a dataset never ends up with two refreshes racing to write it. Because the replacement is whole, send the recipe again even when the only thing you are changing is a dial.

Whether to give it a schedule

Leaving the schedule out is both the default and the cheapest thing you can do. The refresh then runs when a viewer opens a page or dashboard bound to the dataset and the stored data is older than the stale window, and it never runs while nobody is looking — a dataset nobody opened this week costs nothing this week. A cron schedule is the opt-in for data that has to be fresh for something that is not a viewer: a report that emails at eight, an alert that reads the dataset on its own timetable, or a refresh slow enough that you would rather pay for it at seven than make the first person of the day wait for it. The timezone decides what the cron expression means, so set it alongside the schedule rather than leaving 0 7 * * * to mean whichever seven o’clock the server keeps. On-view refreshing stays available beside a schedule, and its three dials are yours: whether a view may trigger a run at all, how stale the data has to be before a view triggers one, and how long after a run another view has to wait — the debounce is what stops a burst of page loads from starting a fresh run each.

What configuring does before it answers

A deterministic refresh — a JavaScript recipe or a script — is run once as part of configuring it, and the outcome comes back on test_run: whether it succeeded, how long it took, and the error text if it did not. Read that field. A configuration that saved cleanly and fails on its first real run is exactly what the test run catches, and the alternative is discovering it when the cron fires at seven the next morning, or when somebody opens the dashboard. A webhook refresh’s response also carries the URL to post to and the signing secret. The secret is returned once, by the configure call that generated it, and never by a read — so capture it when you create the refresh rather than planning to look it up later. Replacing a webhook configuration rotates its signing key and returns the new key once; update the sender to use it. Internal maintenance preserves the existing key without returning it.

Reading a refresh back

Reading a refresh answers with the whole configuration: the strategy and mode, the key column an upsert merges on, its source-write triggers, the cron schedule with its timezone and whether that schedule is armed, the three on-view settings, the recipe itself, its saved parameters, Python script_entrypoint, webhook payload_schema, and the last run’s outcome — when it ran, whether it succeeded, how long it took, and the error text if it failed. When editing a recipe, preserve those values in the replacement. The read names the cron zone schedule_timezone (configure accepts timezone), event references in event_triggers (configure accepts event_dataset_slugs), and context references context_dataset_ids (configure accepts those IDs in context_dataset_slugs). sample_payload is only a configure-time check, so it is not saved or returned. configured: false is an answer, not an error. It means nothing keeps that dataset current — no schedule, no on-view refresh, no live source. A dataset that looks stale and reads back configured: false was never given a refresh, which is a different problem from one whose refresh is failing. A refresh that measured an experiment reads back as disarmed once that experiment has settled. Erdo stops a measurement refresh when the experiment it was measuring is decided or archived, so the flag explains a schedule that has deliberately stopped firing.

When a refresh script keeps failing

The dataset overview shows the latest refresh status above its automatic repair history. Each history item records a repair attempt and keeps its own outcome. A failed attempt can remain in that history after a later refresh succeeds. Use Open automation to inspect the attempts and execution history together. A refresh’s last-run status and error text are always on the refresh itself, but you should not have to go looking. When a script refresh throws or fails to compile on a real run — a bug in its recipe, not a passing timeout or a busy moment — Erdo raises an item in your work feed, tied to the dataset, so the failure is visible without anyone opening a dashboard. It stays open while the script keeps failing and clears itself the next time the refresh runs clean. The item points you at the recipe, because that is where the fix lives: the dataset is serving stale data (or none) because its producer is broken, so editing a page built on the dataset would not help. A refresh that fails for a reason a recipe edit cannot fix — a temporary timeout, a busy moment — is not raised; it just shows in the last-run status until the next run.

Running it once now

Running the refresh starts the configured recipe immediately, whatever its schedule. The call is asynchronous — it answers with the job and job execution it started, the strategy, and a status, and returns before the refresh has finished — so the outcome is read back from the refresh itself, where the last run’s status, error, and duration report a manual run the same way they report a scheduled one. Asking for a run while one is already in progress is refused rather than queued, so two runs can never write the same dataset at once; wait for the one in flight and read its outcome instead.

A nightly rebuild, end to end

A summary table that is expensive to compute and gets read by a report every morning is the case for a cron: it has to be current at seven whether or not anybody has opened the dashboard. replace is right here because the recipe returns the whole table on every successful run.

# @rebuild.js reads the source datasets and returns the complete summary
erdo datasets refresh set pipeline-summary \
  --strategy script_js \
  --mode replace \
  --schedule "0 7 * * *" \
  --transform-js @rebuild.js

# Read back the configuration and how the last run went
erdo datasets refresh show pipeline-summary

curl -X PUT https://api.erdo.ai/v1/datasets/pipeline-summary/refresh \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
        "strategy": "script_js",
        "refresh_mode": "replace",
        "schedule": "0 7 * * *",
        "timezone": "Europe/London",
        "transform_js": "YOUR_RECIPE_JS"
      }'

curl "https://api.erdo.ai/v1/datasets/pipeline-summary/refresh" \
  -H "Authorization: Bearer YOUR_API_KEY"

Both configure calls answer with test_run — the run Erdo made before saving — so a broken recipe is a failure you see now rather than a table that is quietly empty at seven tomorrow. Once it is saved, the read reports the schedule, whether it is armed, and the last run’s status.

CLI

# What keeps this dataset current, and how its last run went (--json for the raw result)
erdo datasets refresh show <dataset-slug>

# Install or replace a refresh. --transform-js, --script and --agent-prompt each
# take an inline value or @path to read the recipe from a file
erdo datasets refresh set <dataset-slug> \
  --strategy script_js \
  --mode replace \
  --schedule "0 7 * * *" \
  --transform-js @refresh.js

# Merge rather than replace: upsert needs the column rows are matched on
erdo datasets refresh set orders \
  --strategy script_js \
  --mode upsert \
  --key-column order_id \
  --transform-js @orders.js

# No --schedule, so it refreshes on view: at most every 12 hours, and never
# within a minute of the last run
erdo datasets refresh set orders \
  --strategy script_js \
  --mode upsert \
  --key-column order_id \
  --transform-js @orders.js \
  --stale-after 43200 \
  --debounce 60

# Let the recipe read other datasets while it builds this one
erdo datasets refresh set pipeline-summary \
  --strategy script_js \
  --transform-js @rebuild.js \
  --context-datasets deals,accounts

# Leave the schedule as the only thing that runs it (--on-view turns viewing back on)
erdo datasets refresh set pipeline-summary \
  --strategy script_js \
  --schedule "0 7 * * *" \
  --transform-js @rebuild.js \
  --no-on-view

# Run the configured refresh once, now
erdo datasets refresh run <dataset-slug>

set replaces the whole configuration, so every call carries the strategy and the recipe — including the ones above that look like they are only changing a dial.

MCP tools

Tool	Description
`erdo_configure_dataset_refresh`	Install or replace a dataset’s refresh. Takes `dataset_slug`, `strategy`, the recipe matching that strategy, and the mode, schedule, and on-view fields below. Answers with the saved configuration, the schedule it created, the strategy it replaced when there was one, a webhook’s URL and secret, and the `test_run` it made before saving.
`erdo_get_dataset_refresh`	Read a dataset’s refresh: strategy, mode, the recipe, the cron schedule and whether it is armed, the refresh-on-view settings, and the last run’s outcome. Read-only, and `configured: false` when the dataset has no refresh.
`erdo_run_dataset_refresh`	Run the configured refresh once, now. Answers with the job it started; the outcome is read back with `erdo_get_dataset_refresh`.

REST

MCP tool	REST endpoint	Method
`erdo_configure_dataset_refresh`	`/v1/datasets/:datasetSlug/refresh`	PUT
`erdo_get_dataset_refresh`	`/v1/datasets/:datasetSlug/refresh`	GET
`erdo_run_dataset_refresh`	`/v1/datasets/:datasetSlug/refresh/run`	POST

The configure body is the worked example above. These are the fields it takes:

Field	What it sets
`strategy`	Required. `script_js` or `script` for a deterministic recipe, `agent_run` for one that needs judgement on every run, `live_webhook` for a streaming source. Native polling is unavailable; use scheduled `script_js` to poll a source.
`refresh_mode`	`replace` or `upsert`. Omitting it means `upsert` when you have named a `key_column` and `replace` when you have not — a key column is only meaningful to a merge.
`key_column`	Required with `upsert`: the column, or comma-separated composite key, rows match on. Whitespace around names is ignored. Upsert requires at least one named column; whitespace-only or comma-only keys are rejected before configuration or script authoring. The CLI also rejects an explicitly empty `--key-column`.
`script` / `transform_js` / `agent_prompt`	The recipe itself. Send exactly the one that matches the strategy.
`script_entrypoint`	The filename assigned to the wrapped Python recipe, default `main.py`. Use a Python module name such as `refresh.py`: ASCII letters, numbers and underscores, beginning with a letter or underscore. Paths, `__init__.py` and names that collide with bundled helper modules are refused. The standard helpers remain available with a custom filename.
`schedule`, `timezone`	A cron expression and the zone it is read in. Omit `schedule` for the on-view default.
`refresh_on_view_enabled`, `refresh_on_view_stale_after_seconds`, `refresh_on_view_debounce_seconds`	Whether a view may trigger a run, how old the data has to be first, and how long after a run before a view can start another.
`context_dataset_slugs`	The other datasets this recipe is allowed to read while it builds its own.
`parameters`	Saved values the recipe reads from `PARAMETERS`. Configuration replaces the whole recipe: omitting this field clears the previous values to `{}`. Include every value the new recipe needs.
`live_write_mode`, `payload_schema`, `sample_payload`	For webhook refreshes: whether an arriving event is appended or replaces current state, the shape the reshaped row must have, and an example payload to check the recipe against before it is saved.

# Read the current refresh — strategy, mode, schedule, on-view settings, recipe,
# and the last run's status, error and duration
curl "https://api.erdo.ai/v1/datasets/pipeline-summary/refresh" \
  -H "Authorization: Bearer YOUR_API_KEY"

# Run it once, now. The response names the job it started; read the outcome back
# from the GET above once the run has finished
curl -X POST https://api.erdo.ai/v1/datasets/pipeline-summary/refresh/run \
  -H "Authorization: Bearer YOUR_API_KEY"

Configuring a refresh or running one needs edit access to the dataset; reading one needs view access. See Data & Connectors for datasets and Automations for other scheduled work.

Reconfiguration and stopped experiments

Configuring a refresh requires edit access to its target and view access to every context dataset. Invalid schedules, unavailable context datasets and rejected recipes leave the existing refresh unchanged. Concurrent configuration calls are serialized per dataset: the recipe, parameters and replacement job link publish together. Future recipe replacements retain the previous automation, its triggers, execution history and repair attempts. The old automation is retired: it cannot be run, edited or rearmed. Batch replacements also retain a link to the replacement before activation is attempted. History remains available by the exact automation ID; it is not combined with the replacement’s history. Previously deleted records are not recovered. Retirement and disabling the old triggers happen atomically before replacement activation. Non-self replacements then withdraw pending approvals and cancel currently linked active agent runs; errors returned by those cleanup owners are reported. A refresh reconfiguring itself keeps its already-running execution so it can finish. Already-started deterministic work may also finish. Runs whose startup link arrives later are not guaranteed to be cancelled by this cleanup. If infrastructure fails after the configuration is saved, the command reports the failure and identifies the old automation when its retirement or run cleanup failed. A batch replacement stays disabled until those steps succeed; its lineage is retained even if activation fails. Repeating the complete configuration retries cleanup of retained predecessors, including after a switch to a live feed or a failed retirement, before activating a new batch replacement. Existing replacement links are preserved; a link lost in a failed transaction is not invented. Invalid, cyclic or oversized predecessor histories fail visibly rather than skipping cleanup. refresh show reports the linked job’s actual enabled state. A measurement refresh bound to a settled or disarmed experiment cannot be rearmed by reconfiguration. Use a new dataset for a new experiment. Updating a still-active experiment’s recipe preserves its binding. Webhook feeds run on incoming events and use live_write_mode; page views and cron do not drive them. Refresh setup checks the current stored CSV header, including columns added by capture since its last analysis. Before replacing a CSV, the writer reads that header again while holding the dataset write lock. If a capture added a column after setup, a refresh that omits it fails and keeps the existing file. A storage read failure also stops the refresh; cached analysis cannot prove that replacing the file is safe.

Get started

Features

Connect Erdo

When refreshes run

Script or agent

Bind an experiment measurement refresh

Trigger a refresh from source writes

Replace, upsert, and no changes

Editing a refresh

Whether to give it a schedule

What configuring does before it answers

Reading a refresh back

When a refresh script keeps failing

Running it once now

A nightly rebuild, end to end

CLI

MCP tools

REST

Reconfiguration and stopped experiments

​When refreshes run

​Script or agent

​Bind an experiment measurement refresh

​Trigger a refresh from source writes

​Replace, upsert, and no changes

​Editing a refresh

​Whether to give it a schedule

​What configuring does before it answers

​Reading a refresh back

​When a refresh script keeps failing

​Running it once now

​A nightly rebuild, end to end

​CLI

​MCP tools

​REST

​Reconfiguration and stopped experiments

When refreshes run

Script or agent

Bind an experiment measurement refresh

Trigger a refresh from source writes

Replace, upsert, and no changes

Editing a refresh

Whether to give it a schedule

What configuring does before it answers

Reading a refresh back

When a refresh script keeps failing

Running it once now

A nightly rebuild, end to end

CLI

MCP tools

REST

Reconfiguration and stopped experiments