Write-Back: Making the Upbeat Sync Two-Way

Design doc — continuation of the drivers arc: importer-drivers-architecture.md → contacts-architecture-crm-as-hub.md → upbeat-contacts-driver-pilot.md → this doc.

Status: PROPOSED · Owner: Glen · Last updated: 2026-07-07

1. Why now

The hub is one-way. @agend/source-drivers ships two ingest drivers (upbeat-contacts, upbeat-sales), a loader, watermark/content-hash delta sync on Inngest crons — and zero write path back to Upbeat. The only trace of two-way is the product-vision comment in the architecture doc:

// bidirectional (product vision): writeBack(normalized) for Agend-owned fields

Two forces make this the next priority:

Client self-development (AHRI). We want clients building on api.agend.com.au instead of receiving our proprietary WordPress plugins. Today that works for reads only. Every transactional flow — join, renew, profile update, pay invoice, event registration — still writes to Upbeat exclusively through the legacy Iugo_Membership_Kiosk_API PHP connector living on the client's own server. Until the hub can write back, we cannot retire those plugins from client environments, and we're stuck protecting them with code encoders as a bridge.
The synchronous-write problem we already diagnosed. The agend-pro upbeat-async-proposal / api-hardening-roadmap work established that synchronous writes to Upbeat during user requests are the platform's biggest fragility (timeouts, partial failures mid-checkout). The hub is the natural fix: accept the write locally (fast), confirm to the user, propagate to Upbeat asynchronously with retries. Write-back isn't just a feature — it's the architectural completion of that hardening arc.

Goal

Agend CRM stays the system of engagement and canonical hub. Upbeat stays the client's system of record for source-owned data. Writes originating in Agend (portal, public API, WordPress spokes) land in the hub first, then propagate to Upbeat reliably, idempotently, and without echo loops — so a client on the public API gets full transactional capability with no Agend code on their server.

2. Current state (verified against code)

Component	State	Evidence
Ingest (Upbeat → hub)	✅ Shipped: contacts + sales	`src/drivers/upbeat-{contacts,sales}/`, `src/loader/{crm,sales}-loader.ts`
Delta sync	✅ Watermark + content-hash skip	`crm_source_sync_state.last_modified_watermark`; `crm-loader.ts` hash compare
Scheduling	✅ Inngest crons (daily delta 03:00, weekly full, monthly sales)	`src/runner/upbeat-orchestrators.fn.ts`
Field ownership	✅ Loader overlays Upbeat-owned fields, preserves CRM-added custom fields/tags	`crm-loader.ts` merge logic; `contacts-architecture-crm-as-hub.md` §field-ownership
Downstream fan-out	✅ `crm.contact.created/updated` webhook events (suppressible: `UPBEAT_INGEST_EMIT_EVENTS`)	`crm-loader.ts`
Write-back (hub → Upbeat)	❌ Absent	No `writeBack` implementation anywhere in the package
Events entity	❌ No Upbeat driver (native Events module is unrelated)	deferred per `upbeat-contacts-driver-pilot.md`

3. The write surface we must absorb

Inventory of write operations the legacy connector (class-imk-api.php) performs against Upbeat's BusinessConnector today, grouped and tiered by criticality. This is the complete contract; write-back is done when the tiers we choose to support are covered.

Tier 1 — profile & account maintenance (low risk, high frequency, natural first phase)

update_member, update_member_specific_values, update_member_single_field
update_account, update_account_specific_values, update_regions_for_account
update_member_specialties, create/update/delete_qualification, create/update_education_history

Tier 2 — commerce transactions (revenue-critical, must be ordered and atomic-ish)

create_invoice, create_invoice_on_account, update_invoice, update_invoice_specific_values
add_product_to_invoice, remove_product_from_invoice, update_product_line, add_option(s), remove_option
update_invoice_payment
create_member_payment_profile, create_payment_profile_for_member, update_member_payment_profile

Tier 3 — membership lifecycle

create_member (join), create_account, create_account_email_domain_association
update_member_subscriptions, update_entitlement
create/update/remove_member_role, create_membership_enquiry, set_password

Tier 4 — events (blocked on the Upbeat events ingest driver)

create_member_registration, update_registration, update_registration_specific_values
create_event_request, update_event_request, create_event_survey
create_event_waitlist_entry, delete_event_waitlist

Tier 5 — engagement records (nice-to-have)

create_case, update_case, create_annotation, create_task, speaker-profile create/update

Everything not listed (log filters, settings mutators) is connector plumbing, not domain writes.

4. Design principles

Hub-first writes. Every write lands in the hub's Postgres synchronously and is confirmed to the caller from there. Upbeat propagation is asynchronous by default. The user never waits on BusinessConnector. (This is the async-proposal lesson, made structural.) Scope clarification (2026-07-08): "async by default" governs mastered records, not request-time computation. Flows where Upbeat is the pricing/computation oracle (e.g. draft-invoice cart pricing returning the real member-specific price) remain synchronous, hub-routed calls outside the outbox — see the Oracle class in upbeat-data-bridge-map.md §1. Do not force quote-style reads through this machinery.
Field ownership is the arbiter. The existing ownership map decides direction per field: Upbeat-owned fields flow in on ingest and are never written back from hub edits (they're read-only in Agend UIs or edits are rejected); Agend-owned fields flow out on write-back and are never clobbered by ingest (already true in the loader). Shared fields (the interesting set: names, email, phone, address) flow both ways under the conflict policy in §7.
Exactly-once effect, at-least-once delivery. BusinessConnector offers no idempotency keys, so we build effect-idempotency on our side: an outbox state machine plus pre-flight verification for creates.
No echo loops. A write-back mutates the record in Upbeat, which bumps dateModified, which the next delta ingest will return. The pipeline must recognise its own writes and no-op (§6).
Fail visible, never silent. Mirrors the legacy connector's purpose-built exceptions (API_Invoice_Not_Created_Exception, …). Every terminal failure produces an operator-visible artefact (dead-letter row + notification), never a swallowed error.
Per-account serialisation. Ingest already runs concurrency-1 per account; write-back dispatch does the same so ingest and write-back for one account never interleave mid-entity.

5. Architecture

                       ┌──────────────────────────────────────────────┐
                       │                 AGEND HUB                     │
 Portal / public API   │                                              │
 / WP spoke            │  1. write lands in hub tables (sync, fast)   │
   ── POST ──────────► │     crm_contacts / invoices / …              │
   ◄── 200/202 ─────── │  2. outbox row enqueued (same txn)           │        ┌───────────┐
                       │     crm_source_writeback_outbox              │        │  UPBEAT   │
                       │  3. Inngest dispatcher (per-account, conc 1) │──────► │ (Dynamics │
                       │     reverse-map → authenticate → POST        │  4.    │  Business │
                       │  5. on success: stamp echo ledger,           │ ◄───── │ Connector)│
                       │     update content_hash expectation,         │        └───────────┘
                       │     emit crm.writeback.succeeded             │
                       │  6. next ingest delta sees own write →       │
                       │     hash matches → no-op (no echo)           │
                       └──────────────────────────────────────────────┘

5.1 Interface: `WriteBackDriver`

Extend the driver contract rather than widening SourceDriver (keeps read-only drivers valid):

/** A single outbound mutation against the external system. */
export interface WriteBackOperation {
  /** e.g. 'contact.update', 'invoice.create', 'invoice.addLine' */
  op: string;
  externalId: string | null;        // null for creates
  /** Agend-side canonical payload (hub field names). */
  payload: Record<string, unknown>;
  /** Rows in the same group execute in order; group halts on failure. */
  groupId: string | null;
  sequence: number;
}

export interface WriteBackResult {
  externalId: string;               // assigned/confirmed by Upbeat
  /** Raw response payload, persisted for the echo ledger + audit. */
  raw: Record<string, unknown>;
}

export interface WriteBackDriver {
  readonly id: string;              // e.g. 'upbeat-contacts'
  /** Which ops this driver can execute, e.g. ['contact.update', 'contact.create']. */
  readonly supportedOps: readonly string[];
  /** Reverse of map(): hub field names → BusinessConnector body. Pure. */
  reverseMap(op: string, payload: Record<string, unknown>): Record<string, unknown>;
  /** Execute one operation. Throws typed errors (Retryable vs Terminal). */
  execute(session: Session, op: WriteBackOperation): Promise<WriteBackResult>;
  /** Pre-flight for creates: does an equivalent record already exist? (dedupe on retry-after-crash) */
  findExisting?(session: Session, op: WriteBackOperation): Promise<string | null>;
}

Auth reuses the ingest machinery untouched: same authenticate(), same crm_source_driver_credentials vault, same crm_source_token_cache.

reverseMap is the mirror of each driver's existing mapping.ts and must live beside it — one file owns a field's name in both directions, so mappings can't drift apart.

5.2 The outbox

New table (migration sketch):

create table crm_source_writeback_outbox (
  id               uuid primary key default gen_random_uuid(),
  account_id       uuid not null references accounts(id),
  driver_id        text not null,                 -- 'upbeat-contacts' | 'upbeat-sales' | …
  op               text not null,                 -- 'contact.update', 'invoice.create', …
  entity_table     text not null,                 -- 'crm_contacts', …
  entity_id        uuid not null,                 -- hub row the write came from
  external_id      text,                          -- null until a create succeeds
  payload          jsonb not null,                -- hub-canonical, reverse-mapped at dispatch
  group_id         uuid,                          -- ordered multi-step transactions (§5.4)
  sequence         int not null default 0,
  status           text not null default 'pending'
                   check (status in ('pending','in_flight','succeeded','failed','dead_letter','superseded','conflict')),
  attempts         int not null default 0,
  next_attempt_at  timestamptz,
  last_error       text,
  idempotency_key  text not null,                 -- hash(account, op, entity, payload) — dedupes client retries
  created_at       timestamptz not null default now(),
  completed_at     timestamptz,
  unique (account_id, idempotency_key)
);

Enqueue happens in the same Postgres transaction as the hub write (transactional-outbox pattern) — a hub write and its propagation intent can never be split by a crash.

Coalescing: a new pending row for the same (entity_id, op) marks older pending rows superseded. Ten rapid profile edits become one Upbeat call carrying the final state.

5.3 Dispatch (Inngest)

Symmetric with ingest:

crm-source-drivers/upbeat.writeback.enqueued → per-account dispatcher function, concurrency 1 per account (shared virtual queue key with the ingest runner so the two never interleave for one account).
Retry with exponential backoff on RetryableError (5xx, timeout, token expiry → refresh and retry); after N attempts (default 8, ~6h span) → dead_letter + operator notification (reuse the ingest failure-notification path).
TerminalError (422 validation, 404 unknown external id) → failed immediately, no retry, notify.
A sweeper cron requeues in_flight rows older than a stall threshold (crash recovery); creates re-run findExisting first so a crashed-after-success create doesn't duplicate.

5.4 Operation groups (the invoice problem)

Click-to-pay is not one write: create_invoice → add_product_to_invoice (×n) → update_invoice_payment. BusinessConnector has no transactions, so we model groups: rows sharing group_id execute strictly in sequence; the first failure halts the group (later rows stay pending, blocked); the group surfaces as a single logical operation in status APIs. Creates inside a group propagate their assigned external_id forward (invoice id → line-item calls).

Payments caveat: the money itself moves through the gateway (eWAY et al.) synchronously as today — write-back only records the settled payment in Upbeat. A dead-lettered invoice.payment group therefore never means lost money; it means Upbeat's ledger is behind, visibly, until an operator replays it. This ordering (charge first, record async) matches the direction the api-hardening work already chose.

5.5 Public API wiring

Gateway routes gain write-through semantics with zero new auth machinery (scopes already exist):

PATCH /v1/crm/contacts/{id}, PATCH /v1/crm/me/profile → hub update + outbox enqueue → 200 with propagation: { status: 'pending', operationId }.
POST /v1/crm/me/orders/{id}/pay, membership purchase → 202 Accepted + operationId for the group.
New: GET /v1/crm/operations/{id} — poll propagation status.
New webhook events for API consumers: crm.writeback.succeeded / crm.writeback.failed (existing @agend/webhooks infrastructure).

Contract note for client docs: the hub read reflects your write immediately; Upbeat reflects it eventually (typically seconds). Clients that need Upbeat-side confirmation subscribe to the webhook or poll the operation.

6. Echo prevention (closing the loop safely)

The subtle problem: our own write bumps dateModified in Upbeat, so the next delta ingest re-fetches the record. Three stacked guards, cheapest first:

Content-hash convergence (primary). After a successful write-back, recompute and store the entity's expected content_hash reflecting the post-write state. When ingest re-fetches the record, the loader's existing hash-skip sees a match → zero writes, zero fan-out events. Costs nothing new; reuses shipped machinery.
Echo ledger (verifier). execute() persists Upbeat's response (with its dateModified) per operation. On ingest, records whose dateModified matches a ledger entry within a tolerance window are marked origin: agend in external_metadata — for observability and as a guard if hash convergence misses (e.g. Upbeat normalises a value server-side so hashes differ).
Field-ownership merge (backstop, already shipped). Even if a record re-ingests fully, the loader only overlays Upbeat-owned fields and preserves Agend-owned ones — an echo can cause a redundant write but not data loss.

Hash-divergence on echo (guard 2 fires but 1 didn't) is logged to crm_source_unmapped_fields-style telemetry: it means Upbeat transformed our value and the reverseMap needs a normalisation rule.

7. Conflict policy (both sides edited the same record)

Window: member updates their phone in the Agend portal while an admin edits the same contact in Upbeat before our write propagates.

Detection: dispatcher compares the record's current Upbeat dateModified (cheap GET, piggybacked on findExisting-style pre-flight for shared-field updates) against the ingest watermark captured when the outbox row was created. Fresher-than-expected + same field(s) touched → conflict.
Resolution by ownership class:
- Agend-owned field → hub wins, write proceeds (Upbeat-side edits to these are illegitimate by contract).
- Upbeat-owned field → should have been rejected at the API layer; if present, drop with telemetry.
- Shared field → do not silently clobber. Row → conflict status, surfaced in an operator review queue (CRM admin UI lists both values, one-click pick). V1 keeps humans in the loop; auto-resolution rules (e.g. member-self-service beats admin edit for contact details) can be layered on once we see real conflict frequency — expected to be rare.

8. Phasing

Phase W1 — Contact profile write-back (Tier 1 subset; the pilot) contact.update (specific-values semantics) via upbeat-contacts driver. Outbox + dispatcher + echo guards + PATCH /v1/crm/me/profile wiring + operations endpoint. This mirrors the highest-volume, lowest-risk legacy write (update_member_specific_values) and proves the whole spine end-to-end. Exit criteria: a profile edit via public API reaches Upbeat < 60s p95; forced echo test produces zero redundant hub writes; kill-switch env (UPBEAT_WRITEBACK_ENABLED, mirroring UPBEAT_INGEST_EMIT_EVENTS) verified.

Phase W2 — Commerce groups (Tier 2) Invoice create/lines/payment as operation groups on upbeat-sales; qualification/education CRUD rides along as simple ops. Unlocks click-to-pay and renewals through the API.

Phase W3 — Membership lifecycle (Tier 3) contact.create (join) with findExisting dedupe, account create/update, subscriptions, entitlements, roles. Unlocks online joining without the WP plugins.

Phase W4 — Events (Tier 4) — blocked on the Upbeat events ingest driver (you can't safely write registrations against events you don't mirror). Sequence the events driver first; registrations/waitlist write-back follows the by-then-proven pattern.

W4 alternative under evaluation — adopt native Events instead of syncing. The dashboard's native Events module may make the events driver (the flagged-hardest, ~40%-bespoke case) unnecessary: migrate event management into Agend, keep only a thin financial up-stroke (registrations → Upbeat invoices via the W2 machinery). Decision framework: upbeat-data-bridge-map.md §4. Do not start W4 driver work before that decision.

Out of scope for all phases: deletes of Upbeat-mastered records (legacy connector barely does this either — only qualifications and waitlist entries; both fold into their tiers), Tier 5 unless a client need appears.

Cross-cutting prerequisite (from the API-surface audit): before the first external client exercises write endpoints — distributed rate limiting on the gateway (current limiter is in-memory per-process) and an RLS backstop or equivalent invariant tests behind the admin-client tenancy model. Writes raise the blast radius of a tenancy bug from "leak" to "corrupt."

9. Observability

Outbox lag (oldest pending age) and dead-letter count per account → existing health-check surface; alert thresholds per tier (Tier 2 pages, Tier 1 warns).
Per-op audit trail: outbox row + echo-ledger response payload = full request/response history per entity (answers "why does Upbeat say X?" forever).
Weekly full-reconcile cron doubles as drift detection: post-write-back, reconcile diffs between hub and Upbeat on shared fields should trend to zero; a nonzero trend is a mapping bug surfaced automatically.
Dashboards: writeback throughput, success rate, p95 propagation latency, conflict rate.

10. Risks

Risk	Mitigation
BusinessConnector semantics differ from legacy PHP assumptions (undocumented field coercions)	Port `reverseMap` rules directly from `class-imk-api.php` bodies (it's the executable spec); drift-detection reconcile catches the rest
Upbeat outage causes outbox pileup	Backoff + dead-letter caps; hub remains fully functional (that's the point); backlog drains on recovery in order
Echo guards miss → event storm to spokes	Flood-guard already exists (`UPBEAT_INGEST_EMIT_EVENTS`); echo-ledger telemetry catches divergence before it's systemic
Duplicate creates on crash retry	`findExisting` pre-flight + idempotency-key uniqueness + sweeper re-verification
Conflict queue ignored by operators	Alert on queue age; conflicts expected rare — if not rare, that's a field-ownership modelling error to fix, not a queue to grow

11. What this unlocks

Each phase retires a slice of the proprietary WordPress plugins from client servers:

W1 → profile/self-service flows move to the API; client-built "edit my details" pages become possible with zero Agend code on the client box.
W2 → click-to-pay, statements, renewals through the API; the Products-Extension plugin's core reason to exist starts dissolving.
W3 → online joining via API; the membership lifecycle no longer requires our plugins.
W4 → events registration parity; the last transactional dependency on the legacy connector is gone. Code-encoding of client-deployed plugins (the ionCube bridge) becomes unnecessary because there is nothing left to protect.

End state: any client (AHRI first) builds their entire member experience on api.agend.com.au + the @agend/elementor-widgets bundle. Upbeat stays their system of record; Agend is the engagement layer; our IP never leaves our infrastructure.