Skip to main content

​
Event pipelines

An event pipeline is a deterministic flow that receives an inbound event and does fixed work with it: a lead-form submission, a third-party webhook, or an in-app action arrives at the pipeline’s endpoint, gets normalized, and runs through an ordered list of actions — write the row to a dataset, send an email, call a connected integration, broadcast a live update, or return a custom response. Pipelines are how a page captures leads, and how Erdo enriches and routes them without an agent in the loop on every event. Each pipeline has:
  • a slug and name — a stable, human-readable handle (e.g. acme-leads-a1b2c3d4),
  • a state — active, disabled, or proposed,
  • a source kind — browser_form, third_party_webhook, custom_http, internal, …,
  • an inbound endpoint URL — where events are POSTed,
  • an auth mode — how inbound events are verified (HMAC, shared secret, provider signature, signed-in viewer, or public),
  • the actions it runs and the datasets it writes to, and
  • an execution log — every event it received, with status (accepted / rejected / failed), a payload preview, the response, and any error.

​
Where data lands

A pipeline’s dataset.write actions persist rows to a dataset — per-row, SQL-queryable storage. To see captured leads, query the dataset the pipeline writes to (shown as write_target_datasets), not the pipeline itself. The pipeline’s execution log tells you whether events are arriving and being accepted; the dataset holds the rows. A failed storage write stops the execution before later actions or a success response, even if that write sets continue_on_error. Optional side effects can still continue on error after capture succeeds. Marked test and health-probe submissions remain stored for verification; their JSON object responses explicitly set conversionEligible: false so they cannot authorize an advertising conversion. Datasets enrolled in permanent lead identity use a canonical capture contract to resolve contacts and write the lead atomically. The permanent lead reference is separate from the conversion transaction ID. Enrollment, producer adoption and backfill happen together in one operator call; an ordinary pipeline edit does not migrate historical rows.

​
Preparing dataset columns before the first event

When you create or update an enabled pipeline, provide sample_payload with a representative event body. The transform runs without executing actions or writing a customer row. Its output is checked against existing target contracts. For a durable dataset with no stored file and no declared schema, that output initializes the columns before the first real event. Platform purpose columns and any enrolled identity key are included automatically. Include every optional output column in the sample. Dynamic output shapes need an explicit dataset schema. Existing files and contracts are preserved; samples cannot widen them. For an unwritten purpose-bound target that has no complete contract, an enabled create, re-enable, or producer change requires a sample that produces normalized rows, or a complete schema declared beforehand. A legacy metadata-only edit can continue but returns a setup warning. Cloning an enabled source with such an incomplete target is refused until the source is configured. Disabled pipelines and unbound datasets can retain deferred schema discovery. The sample must also satisfy request_schema. A revision becomes visible only after its target contracts are ready; a refused registration leaves the previous revision active. Concurrent edits abort instead of overwriting a newer revision. Widget and conversation lead producers establish their fixed output contracts through the same registration before becoming active. An existing built-in also establishes a missing contract on lookup even when its configuration is already current, without changing its pipeline revision. Assigning a partial or custom purpose to an unwritten durable dataset requires a complete declared producer schema. You can declare the schema and set its purpose in one schema update; the declaration is composed with the requested purpose and settled before that purpose is published. Clearing a purpose with a wholesale declaration does not re-add columns belonging only to the previous purpose.

​
Emails a pipeline sends

A pipeline that emails on an event — a lead alert to your sales inbox, a confirmation back to the person who submitted the form — has Erdo compose the message. The pipeline supplies the content (a heading, the captured fields as label/value rows, a button through to the page the lead came from) and Erdo renders it in the standard Erdo styling, with a plain-text version alongside for clients that block HTML. Addresses and links in the fields become clickable, so a lead alert can be replied to from the notification itself.

​
Pipeline purpose

Every pipeline declares a purpose — its role — so you can tell capture flows apart from analytics infrastructure without guessing from names:
  • record_capture — a pipeline whose job is persisting submitted records to a dataset (it has at least one dataset.write action). This is how you find which datasets receive a flow’s records: filter pipelines by purpose = record_capture and read their write_target_datasets (each entry is a fully-qualified org-slug.dataset-slug). You can set this explicitly when creating or updating a pipeline; a custom pipeline that writes to a dataset is also promoted to record_capture automatically at save time.
  • page_events — the managed first-party page-analytics pipeline described below. Reserved for Erdo’s own provisioning; you can’t set it.
  • custom — everything else: a webhook or enrichment flow that doesn’t persist records.
To discover which dataset a campaign’s leads land in, list your record_capture pipelines and read their write_target_datasets, then query that dataset. Every dataset.write row is stamped at ingestion with columns your transform doesn’t set: variant, artifact_id, page_url, and optional session_key (from the page envelope), plus request signals that let you keep your metrics clean and segment them by who actually showed up:
  • session_key — the opaque browser-session key shared by page events and form captures. Join it with the page and time window to follow a visit through to its saved lead. The shared page runtime supplies it automatically; custom page forms using erdo.submitEvent need no extra field or transform mapping. It is internal correlation metadata, never authentication or a person’s identity. Canonical lead merges retain the first recorded session; a later visit does not replace it. For an older lead gaining a key on a later capture, that key does not identify its original acquisition visit. Session storage preserves the key within the tab when available; blocked storage limits continuity to the current document. Missing or invalid optional keys do not reject submissions. Historical captures without a key remain unknown.
  • is_test — a boolean, always present. true when the submission was tagged as test traffic (see Marking test traffic below), otherwise false. Filter it out to count real leads and conversions.
  • device_type — "mobile", "tablet", "desktop", or empty, classified from the request’s User-Agent at ingestion. Segment funnels by it to compare mobile vs desktop (a page can take most of your spend yet convert zero mobile visitors — this column surfaces that without parsing User-Agents by hand).
  • browser — the coarse browser family ("chrome", "safari", "firefox", "edge", "samsung", "opera", "other", or empty), also from the User-Agent.
  • in_app — the in-app webview container the visit came from ("instagram", "facebook", "tiktok", and similar), or empty for a normal standalone browser. In-app webviews routinely break modals and payment flows, so this separates “converts badly on mobile” from “converts badly inside Instagram’s browser”.
  • language — the visitor’s preferred language, the primary tag of the Accept-Language header (e.g. "pt-BR"), or empty. Check that a Brazilian, Colombian, or French campaign is actually reaching PT/ES/FR speakers.
  • country, region, city — the visitor’s coarse location: ISO-2 country (e.g. "US"), ISO-3166-2 region code (e.g. "NY"), and city name, from the serving edge. Empty when the request reaches the pipeline through an ingress that carries no edge geo. Segment by them to verify geo-targeted campaigns (e.g. “New Yorkers relocating to Miami”) landed where you paid for.
  • utm_source, utm_medium, utm_campaign, utm_term, utm_content, utm_id, gclid, gbraid, wbraid, fbclid, li_fat_id — the campaign the visitor arrived from, read from the query string of the page they submitted on. This is what makes a per-campaign cost-per-lead a plain GROUP BY utm_campaign over your leads dataset, and what lets a lead join to the pageview that produced it — page events carry the identically-named columns. gclid is the Google click id, needed to report the conversion back to Google Ads; gbraid and wbraid are what Google sends instead on iOS traffic with app tracking restricted. A column stays empty when the visit carried no such parameter, which is what direct and organic traffic looks like.
These are stamped only where your transform left the column unset, so a page that maps its own column of the same name always wins. Each is epoch-bound — rows written before the column existed carry it blank, which means “no signal”, not a real value like desktop or zero visitors. Campaign values are limited to 512 characters, including values copied by your transform. Older leads may have no campaign columns if their transform did not copy them; the submitted URL is still on the row as page_url, so you can read the attribution back out of it — regexp_extract(page_url, 'utm_campaign=([^&#]*)', 1). utm_id and li_fat_id carry LinkedIn attribution: LinkedIn’s URL tracking appends utm_id beside the other UTMs, and LinkedIn adds li_fat_id to ad clicks. utm_id is kept exactly as the URL supplied it, as text (leading zeroes survive), and is never interpreted as a particular campaign or ad id. On a lead that is captured more than once, each campaign column keeps its first non-empty value, so a later visit can fill a missing id but never replaces one already recorded. Rows captured before these columns existed are not backfilled; their page_url still holds the raw values.

​
Platform columns and declared schemas

For file datasets with a declared schema, capture writes add the platform columns present in that batch as optional columns. This includes request metadata, campaign attribution, widget analytics dimensions and conversation lead fields. Your existing column types, required fields and qualification rules are preserved. Other unexpected transform columns still appear as schema drift; captured rows are retained. Datasets without a declared schema keep their existing behavior. This automatic extension applies to file datasets; external database tables retain their existing schema rules.

​
Passive observations and optional form protection

Browser-form captures retain passive observations for investigations without changing submission acceptance or assigning a bot score. For selected public forms, you can separately enable Managed Turnstile protection. It is off by default per pipeline and does not rank leads by IP address. Protected captures record turnstile_status as verified or unavailable when verification succeeds or the server observes a verification outage. Missing or invalid tokens require verification before capture. These statuses are set by the server; a form or transform cannot supply a trusted verification result.

​
Marking test traffic

QA checks and automated evaluations submit to the same pipelines your real visitors do, so their rows would otherwise inflate lead and funnel counts. Tag a submission as test traffic and it lands with is_test = true, keeping it out of your real-traffic metrics. Two equivalent markers:
  • Send the request header X-Erdo-Test: 1 (1 or true), or
  • Include a top-level "is_test": true field in the JSON body.
The envelope’s marker always wins over the transform, so a transform can never clear a test mark. The marker is trusted as-is: mismarking real traffic as test only hides it from your own dashboards, so there’s no signature to manage. Auto-provisioned page-events datasets ship with a default filter that hides is_test = true rows, so their funnel counts exclude test traffic automatically; add the same is_test not_equals true default filter to your own lead datasets to get the same behaviour on reads.

​
Auto-provisioned page-events pipelines

Alongside the pipelines you or your agents create, every published page gets one page-events pipeline automatically (its purpose is page_events — see Pipeline purpose above). It receives the page’s first-party analytics events — pageviews, scroll milestones, section views, CTA clicks, form starts, leads, video plays — and writes them to your organization’s shared Page events dataset, stamped with the page, experiment variant, session, campaign attribution, is_test, device_type, browser, in_app, language, and geo (country/region/city). Real-user LCP, INP, and CLS measurements arrive as web_vital rows with vital_name, vital_value, and vital_rating. The server also distils recorded sessions into scroll_bail, rage_click, and form_abandon rows with redaction-safe typed dimensions; these names cannot be submitted through the public browser endpoint. Events outside that fixed vocabulary are rejected, which you can see in the pipeline’s execution log. Disabling a page-events pipeline turns first-party analytics off for that page; Erdo won’t re-create it. See Pages → First-party page events for what’s captured and how to query it. The generated page_events action and its shared Page events dataset target are reserved: updates may change ordinary metadata or disable the pipeline, but cannot remove, duplicate, or redirect that write. Trusted replay receipts fail closed unless exactly one keyed write succeeds against the canonical dataset, so a response-only, zero-write, wrong-target, or multiple-write configuration cannot be recorded as settled.

​
Managing your pipelines

You can list your pipelines, read one’s configuration, review its run history, create new pipelines, update their editable fields, and rotate their verification secrets over MCP, the REST API, and the CLI. Everything is org-scoped — a caller only ever sees their own organization’s pipelines — and pipelines are referenced by slug (a UUID is also accepted).

​
CLI

See lead form protection for the configuration file and Cloudflare setup. REST updates and erdo_update_event_pipeline accept the same object in the optional top-level turnstile field. Omission preserves the current policy; an enabled update may omit secret to preserve its encrypted value. The CLI authenticates with your Erdo token and the active organization (erdo login, erdo org).

​
MCP tools

Identity and RBAC ride the request context, so a caller can only reach their own org’s pipelines. Signed-in pages can submit to active authenticated_viewer browser-form pipelines with the record_capture purpose that belong to that page. The page viewer can discover eligible IDs without seeing pipeline configuration, targets, auth settings or secrets; full configuration still requires edit access. The server checks the current pipeline settings again when a workspace submission arrives. Workspace analytics remain suppressed.

​
REST

Base URL https://api.erdo.ai. Authenticate with Authorization: Bearer <token> and select the org with X-Organization-ID. To confirm a pipeline is capturing events end-to-end, check its executions here, then query the dataset it writes to.

​
Marking failures handled

After investigating and dealing with failures, record what you did so health reports can exclude those executions from unresolved failure counts:
The cutoff includes executions received at or before that time and defaults to now. To select specific executions, repeat --execution <id> instead. When IDs are supplied, they take precedence over the cutoff. The command returns {"marked": 12} with the number newly handled. The REST body accepts note, an optional RFC3339 through timestamp, and an optional execution_ids array. MCP exposes the same inputs with pipeline_slug through erdo_mark_event_pipeline_failures_handled. Notes are required, trimmed, and limited to 500 characters; at most 1000 execution IDs are allowed, and future cutoffs are refused. The executions list includes handled_at, handled_by, and handled_note on handled failures. The original outcome and error remain in the history. Accepted executions are never marked, and marking an already-handled execution preserves its first audit entry. Handling a failure does not retry it. Identity refusals with failure_stage: "lead_resolution" are never marked, by cutoff or by ID: they are submissions waiting for a decision, settled through lead-capture dispositions.

​
Reading an execution’s failure_stage

Each execution carries a failure_stage naming the step its outcome was decided at — transform, lead_resolution, dataset_write, write_target, auth, request_validation, state_gate, verification, execute or schema_contract. Read it together with status rather than matching the text of error, which is prose and changes.

​
Creating a pipeline

POST /v1/event-pipelines (or erdo_create_event_pipeline) takes:
  • name (required) — the slug is derived from it,
  • state — active, disabled, or proposed (the default); pass active to start capturing immediately,
  • source_kind — browser_form, third_party_webhook, custom_http (default), …,
  • auth_mode and auth_config — how inbound events are verified. hmac_sha256 (the default), shared_secret_header, and provider_signature carry a verification secret: omit it and one is generated for you. authenticated_viewer accepts signed-in page viewers — set auth_config.audience to "public_and_viewers" for a public lead form,
  • request_schema — a JSON Schema the inbound payload must satisfy,
  • transform_js — function transform(event) { … } normalizing the payload before the actions run,
  • pipeline — the ordered actions array, e.g. [{"type": "dataset.write", "dataset_slug": "riverview-leads"}]. Reference datasets by slug, never UUID — a plain slug targets your own organization’s datasets, and a well-formed slug that doesn’t exist yet is auto-created as the pipeline’s write target (reported back in created_datasets), so provisioning a new campaign’s lead capture is one call,
  • purpose — custom or record_capture (see Pipeline purpose); page_events is reserved,
  • owner_artifact_id — the page this pipeline belongs to, for page-fed pipelines.
The inbound source and auth are set at creation and are not editable afterwards; everything else can change via PUT /v1/event-pipelines/:slug.

​
Save-time warnings

The create and update responses carry a warnings array — advisory findings the save does not refuse but you should act on. Empty means no findings. The one warning today catches a split leads dataset: a pipeline that captures leads by email upsert while other active lead-capture pipelines in your organization write to different datasets. That shape usually comes from two build sessions each minting their own leads dataset (a slug collision suffix like -6894 is the tell), and it silently strands leads — every page looks healthy, but dashboards and email dedupe see only the fraction landing in their dataset. When you see it, point every lead-capture pipeline for one customer at a single leads dataset (an update repointing the dataset.write step), or disable the stale pipelines.

​
Secrets are shown once

For the secret-bearing auth modes, the create and rotate responses carry the plaintext verification secret in a secret field — exactly once. List and get never show it again (the stored auth_config is redacted on every read), so store it when you receive it. Rotating (POST /v1/event-pipelines/:slug/rotate-secret) mints a fresh secret and invalidates the old one immediately — update the sender’s signing configuration right away. Pipelines whose auth mode carries no secret (authenticated_viewer, public_artifact, none) reject rotation.