# Fence custom-event lifecycles and delivery Implement durable epochs, redaction fencing, immutable payloads and bounded attempt accounting in the sending app. ## What you need An authenticated app backend, durable PostgreSQL lifecycle/outbox storage and bounded worker execution. These are implementation recipes for the sending app, not an outbox service provided by HeyCrust. The [API contract](/docs/developers/backend-events) remains the authoritative ingestion behavior. ## Steps 1. Store a durable per-app/shop epoch and monotonic redaction boundary. Separate authenticated reenrollment from recording and delivery. 2. Capture requestStartedAt before authentication; persist the original request time and enrolled epoch with deferred work. Never regenerate them when a queue drains. 3. Serialize record, delivery, cleanup and reenrollment using the same bounded database fence in every process. 4. Reserve and commit an attempt before HTTP, then commit the receipt independently. Keep serialized payload and occurredAt immutable. 5. Cancel and fence work before successful redaction; verify cleanup and any authorized remote erasure. Require a later authenticated request to reenroll. 6. Execute lifecycle, multiprocess and attempt-storage failure regressions locally. Use [dry-run validation](/docs/mcp/validate-custom-events) and inspect exact natural receipts after authorized release. ## Durable lifecycle and identity Maintain a durable app/shop lifecycle row: opaque epoch, active/redacted phase, monotonic lastRedactedAt; store the original epoch with every queued job/outbox event. Retain the minimal redaction fence under a documented privacy retention policy; a domain hash is pseudonymous, not anonymous. Recording/delivery/background cleanup must never create or reenroll lifecycle state. Capture requestStartedAt synchronously at handler entry BEFORE awaiting authentication. Compare clocks conservatively: use one authoritative time domain, or a validated maximum app-to-database clock-skew bound and require requestStartedAt minus that bound to be strictly later than the DB redaction boundary. Fail closed when the bound is unavailable or exceeded; raw process-clock vs DB-clock comparison is unsafe. A DB timestamp sampled after an awaited/delayed query cannot replace the original request start. An old request whose authentication finishes after redaction remains old. Reenroll only via a later authenticated installation/session passing that conservative boundary check and verified current authorization; mint a fresh epoch and keep the boundary. A webhook, timer, queue item, theme check or stale session row cannot reenroll. Event ID example: sha256(JSON.stringify([appId, shopDomain, epoch, name, configurationRevision, publishedThemeId, enableGeneration])). Use a random persisted save/transition UUID or monotonic revision as appropriate, exclude secrets. Reinstall gets a new epoch; changed configuration/theme gets new context; disable→re-enable increments enableGeneration even if theme/config are unchanged. A retry keeps the original event ID, occurredAt and payload bytes. Request-start time belongs to the incoming request, before the first await of authentication. themeCheckedAt instead records when a published-theme verification completed; it cannot authorize reenrollment or replace that early timestamp. After any asynchronous save/theme/browser work, recheck the original epoch, request boundary and current configuration/theme/enable generation under the fence. A stale job must be dropped even if a new installation is now active. An explicit authenticated enrollment adapter returns an epoch token. Recording accepts that token and never creates lifecycle rows. Mint new context for reinstall, changed configuration, changed published theme and each disable-to-re-enable transition. Keep unchanged retries on their original token, ID, timestamp and bytes. ## Shared database fence and redaction All processes use the same per-app/shop database fence for recording, reservation, HTTP delivery, cleanup, redaction and reenrollment. Bound acquisition, connection/pool wait, statements, transactions and HTTP+body. Recheck phase, epoch, boundary and revision under the fence before writes and immediately before HTTP. A process-local mutex/cancel flag is insufficient. Use a dedicated PostgreSQL connection with a session advisory lock and bounded pg_try_advisory_lock polling, or an equivalent proven distributed fence. Keep the fence held across independent reservation commit, HTTP+ack body, and receipt commit. No HTTP starts until its reservation is committed. Never place attempt increment and receipt write in one transaction that can roll back after HTTP. Use a stable collision-resistant lock key derived from app plus shop. Pin one dedicated connection; bound acquisition with pg_try_advisory_lock and a deadline, and configure connection/pool/statement waits. Release the session lock in finally; if unlock/connection state is uncertain, destroy the connection rather than returning a potentially locked session to the pool. Transaction-level advisory locks expire on commit and cannot protect an independently committed reservation followed by HTTP. Redaction: cancel local queued/in-flight tasks, acquire the shared DB fence, set phase=redacted and rotate epoch, advance lastRedactedAt using GREATEST(previous boundary, trusted current cleanup time), erase outbox/config/theme tracking and stale authenticated sessions, commit, then perform any authorized remote erasure. Retain a minimal unresolved-delivery/remote-erasure marker separately from erasable payloads. Complete/retry every cleanup stage and disclose any remote uncertainty before reporting success. Do not delete the fence or reset generations to zero. Reordered cleanup calls cannot move the boundary backwards. The following SQL is an adapter sketch executed while the common session fence is held. Table and column names belong to your sending app; supply bound parameters. The trusted boundary is captured at cleanup under the fence, not from a webhook's historical timestamp. ```sql BEGIN; INSERT INTO tracking_lifecycle (shop_key, epoch, phase, last_redacted_at) VALUES ($1, $2, 'redacted', clock_timestamp()) ON CONFLICT (shop_key) DO UPDATE SET epoch = EXCLUDED.epoch, phase = 'redacted', last_redacted_at = GREATEST(tracking_lifecycle.last_redacted_at, EXCLUDED.last_redacted_at); DELETE FROM tracking_outbox WHERE shop_key = $1; DELETE FROM tracking_theme_state WHERE shop_key = $1; -- Erase stale authenticated session/contact tracking in your app's own schema. COMMIT; ``` Retain the minimal lifecycle fence. Under that same lock, an authenticated enrollment can change redacted to active only when its ORIGINAL requestStartedAt passes the conservative clock-skew-adjusted comparison against lastRedactedAt and authorization is current. Generate a new random epoch and retain lastRedactedAt. Queue items, ordinary saves, theme probes and worker timers cannot perform this transition. Cleanup failure is retried and cannot be reported as successful redaction. Abort cannot retract an HTTP request already accepted remotely. A fence orders local calls that complete while it is held and prevents new local sends after cleanup; it does not order remote completion of timed-out calls. Any transport/body timeout, lost acknowledgement or DB fence loss after HTTP starts is uncertain. Persist that uncertainty with the committed reservation and retain a minimal unresolved marker through local payload erasure. Retry only identical bytes while the same lifecycle remains active. Redaction must cancel retries and report remote finality incomplete until authorized reconciliation can establish it. Current /api/events has no receiver-side lifecycle fence or erasure tombstone; neither an abort, a later receipt nor DELETE alone proves no late remote write. Strong remote redaction finality under unresolved network requests requires receiver-side support and cannot be promised by these sender examples. ## Durable attempts and bounded HTTP Recommended sender policy (not API limits): at most six durable reservations, 24-hour pending age, bounded queue/batch size, two-second HTTP+body deadline, 8 KiB ack limit, bounded DB fence. Reservation CAS requires pending/current epoch/attempt\<6 and commits attempts+1 plus nextAttemptAt before HTTP. Crashes after reservation conservatively spend the attempt. Receipt-storage failure never restores it. A provable pre-HTTP budget skip may CAS-release its reservation once; after HTTP starts never refund it. Reconcile abandoned reservations and mark exhausted/expired rows terminal without another send. A session fence must remain held while the reservation transaction commits. Use a one-time reservation UUID and compare-and-swap on epoch, pending state, attempts and nextAttemptAt. Reject an expired age or six attempts before HTTP. A committed reservation is conservatively consumed after a crash; an explicit budget skip that proves HTTP never started can release only its own reservation once. Do not decrement after an HTTP error or failed receipt write. Persist retry state and nextAttemptAt with capped exponential backoff plus jitter. Interpret Retry-After as seconds or HTTP date. Do not retry before a longer requested delay; expire/pause instead if it exceeds the sender's age budget. Schedule a bounded number of events per tick and retain a durable queue, not just an in-memory timer. ## Shared Node worker The following executable HTTP/worker example uses application-specific database adapters. Implement every adapter using the protocol above before enabling delivery. No actual database adapter is supplied by this guide. ```javascript async function deliverOne(store, job, key, url, signal) { // withShopFence holds the SAME bounded cross-process DB lock as record/redact. return store.withShopFence(job.shopDomain, async fence => { const current = await fence.loadPending(job.eventId); if (!current || !await fence.isCurrentActiveEpoch(current)) return; if (fence.remainingMs() < 2500 || signal?.aborted) return; // no attempt // Commits independently BEFORE HTTP, while the session fence stays held. const reservation = await fence.reserveCommitted(current, { maxAttempts: 6, maxAgeMs: 86400000 }); if (!reservation) return; if (fence.remainingMs() < 2500 || signal?.aborted) { await fence.releaseNoHttpReservationCommitted(reservation); // CAS; only before HTTP return; } const result = await postBounded(url, current.payloadBytes, key, signal); // Separate commit. Persist uncertain outcomes; they block remote-redaction finality. // Failure here must NEVER roll back the reservation or erase its unresolved marker. await fence.saveReceiptCommitted(reservation, result); }); } async function postBounded(url, payloadBytes, key, signal) { if (signal?.aborted) return { state: 'cancelled' }; // no HTTP starts let events; try { const payload = JSON.parse(payloadBytes); events = payload.events ?? [payload]; if (!Array.isArray(events) || !events.length || events.length > 50 || !events.every(event => typeof event.appId === 'string')) return { state: 'failed' }; } catch { return { state: 'failed' }; } // malformed local payload; no HTTP const controller = new AbortController(); const cancel = () => controller.abort(); signal?.addEventListener('abort', cancel, { once: true }); if (signal?.aborted) controller.abort(); let timer; const timeout = new Promise(resolve => { timer = setTimeout(() => { controller.abort(); resolve({ state: 'uncertain' }); }, 2000); }); try { return await Promise.race([timeout, (async () => { try { const response = await fetch(url, { method: 'POST', redirect: 'manual', signal: controller.signal, headers: { 'Content-Type': 'application/json', Authorization: 'Bearer ' + key }, body: payloadBytes }); if (!response.ok) { await response.body?.cancel(); return { state: [408, 429].includes(response.status) || response.status >= 500 ? 'uncertain' : 'failed', status: response.status, retryAfter: response.headers.get('Retry-After') }; } const reader = response.body?.getReader(); if (!reader) return { state: 'uncertain' }; const decoder = new TextDecoder(); let bytes = 0, body = ''; try { for (;;) { const chunk = await reader.read(); if (chunk.done) break; bytes += chunk.value.byteLength; if (bytes > 8192) { await reader.cancel(); return { state: 'uncertain' }; } body += decoder.decode(chunk.value, { stream: true }); } body += decoder.decode(); const ack = JSON.parse(body); if (ack?.ok !== true || !Number.isInteger(ack.stored) || ack.stored < 0 || !Number.isInteger(ack.deduped) || ack.deduped < 0 || ack.stored + ack.deduped !== events.length || ack.scope !== 'app' || !Array.isArray(ack.results) || ack.results.length !== events.length || !ack.results.every((result, i) => result?.appId === events[i].appId && result.scope === 'app' && typeof result.id === 'string' && /^[0-9]+$/.test(result.id))) return { state: 'uncertain' }; return { state: 'acknowledged', receipt: { ok: true, stored: ack.stored, deduped: ack.deduped, scope: ack.scope, results: ack.results.map(({ appId, scope, id }) => ({ appId, scope, id })) } }; } finally { reader.releaseLock(); } } catch { return { state: 'uncertain' }; } // HTTP started: abort/error cannot prove non-acceptance })()]); } finally { clearTimeout(timer); controller.abort(); signal?.removeEventListener('abort', cancel); } } ``` Examples illustrate call sites and the Node sender, not a drop-in database SDK. Implement acceptAuthenticatedSession/accept_authenticated_session as explicit bounded non-fatal enrollment returning an epoch token only after current authorization and conservative clock-skew-adjusted original request-start boundary checks under the shared fence. recordCommittedSave/record_committed_save must be bounded non-fatal adapters that preserve that token, validate it under the fence and never reenroll. store.withShopFence must pin one DB connection, use the same lock key everywhere, bound waits/statements, and retain its session lock across separate transactions. reserveCommitted uses CAS, persists nextAttemptAt and commits before returning; releaseNoHttpReservationCommitted is a one-time CAS; saveReceiptCommitted retains attempts and the unresolved reservation marker on failure, persists the validated receipt on acknowledgement, and stores uncertain outcomes plus bounded retry scheduling with capped exponential jitter and Retry-After. Uncertain attempts remain distinct from acknowledged delivery and must not be forgotten during redaction. Request handlers may not use unawaited promises on serverless; use durable queues/platform lifetime support. Transaction-only advisory locks cannot span these independent commits. Never copy these named adapters as empty stubs or rely on in-memory counters. ## Framework call sites Each example assumes a bounded non-fatal tracking adapter: optional tracking failure must not reverse a committed business save or fail its response. Ensure an enclosing settings transaction really committed before recording; save! or a nested transaction alone does not establish that fact. ### Remix / Shopify React Router ```javascript export async function action({ request }) { const requestStartedAt = new Date(); // BEFORE authentication; adapter applies validated clock-skew bound const { session } = await authenticate.admin(request); const lifecycle = await tracking.acceptAuthenticatedSession({ shop: session.shop, requestStartedAt, authenticatedSession: session }); // explicit bounded/non-fatal enrollment; returns epoch token const saved = await saveSettings(request, session); // must commit first if (lifecycle && saved.changed) await tracking.recordCommittedSave({ shop: session.shop, requestStartedAt, epoch: lifecycle.epoch, revision: saved.revision, appId: "11111111-1111-4111-8111-111111111111" }); // Adapter calls are bounded/caught so optional tracking cannot break the save. // Drain through a durable worker; on serverless use platform waitUntil/queue. return saved.response; } ``` ### Express ```javascript app.use((req, res, next) => { req.trackingStartedAt = new Date(); next(); }); // BEFORE auth; adapter applies validated clock-skew bound app.post('/settings', authenticateShop, async (req, res) => { const lifecycle = await tracking.acceptAuthenticatedSession({ shop: req.shopSession.shop, requestStartedAt: req.trackingStartedAt, authenticatedSession: req.shopSession }); const saved = await saveSettings(req, req.shopSession); if (lifecycle && saved.changed) await tracking.recordCommittedSave({ shop: req.shopSession.shop, requestStartedAt: req.trackingStartedAt, epoch: lifecycle.epoch, revision: saved.revision, appId: "11111111-1111-4111-8111-111111111111" }); res.json(saved.response); }); // bounded non-fatal tracking adapter + durable worker; same shared DB fence ``` ### Rails ```ruby prepend_before_action :capture_tracking_start before_action :authenticate_shop! def capture_tracking_start @tracking_started_at = Time.now.utc # BEFORE auth; adapter applies validated clock-skew bound end def update lifecycle = Tracking.accept_authenticated_session(shop: current_shop, request_started_at: @tracking_started_at, authenticated_session: current_session) # Outer transaction must be committed before recording; nested save! is insufficient. saved = SettingsService.commit_update!(current_shop, permitted_settings) Tracking.record_committed_save(shop: current_shop, request_started_at: @tracking_started_at, epoch: lifecycle.epoch, revision: saved.revision, app_id: "11111111-1111-4111-8111-111111111111") if lifecycle && saved.changed? render json: saved.response end # Tracking is bounded/non-fatal; ActiveJob worker implements the SAME DB fence protocol. # Net::HTTP: max_retries=0; open/read/write timeouts plus an overall HTTP+body deadline, # 8 KiB acknowledgement cap and no redirect following. Accept only 2xx + JSON ok == true. # Reserve attempts in an independently committed transaction BEFORE HTTP; # receipt writes use a later transaction. Do not wrap both in ActiveRecord::Base.transaction. ``` For serverless request lifetime, use a durable queue and the platform's lifetime primitive when persisting work after the response. For long-running processes, timers may wake the durable worker but cannot be the sole retry record. Framework hooks must capture the request timestamp before all auth middleware, not after an after-auth hook begins. ## Retention and rollback Set and implement an explicit local retention period for pending/delivered/failed payloads and receipts. Disable tracking on rollback; stop workers and cancel queued work, preserve epochs/boundaries/attempt counts through schema/code rollback. Do not replay old outboxes on restart. Rollback cannot undo remote events/messages. On redaction erase payloads and bounded logs; retain only the lawful minimal fence needed to reject stale work. There is no universal supported local retention duration: this guide supplies a 24-hour pending budget as a recommended sender policy. Define delivered/failed record purge periods explicitly in your app. Deleting a local payload does not delete the remote event; replaying it after remote erasure can recreate tracking. Pausing a sender does not undo a previously accepted event or message. ## Expected result A sender that rejects stale epochs/boundaries, records committed successful milestones, preserves immutable retry bytes and cannot exceed its durable attempt budget when receipt storage fails. A validated payload or local test is not a live API, Flow or message receipt. ## Troubleshooting Cover at least queued work after redaction, delayed authentication crossing the boundary, reordered cleanup, reinstall ID reuse, configuration/theme changes, disable/re-enable generations, two workers sending one occurrence, receipt failure after HTTP, six-attempt exhaustion, pre-HTTP budget skips and HTTP/body deadlines. Inspect [exact natural receipts](/docs/mcp/get-custom-event-receipt) after an authorized release; do not create synthetic events to make a check pass. Report live milestone, event receipt and Flow/message verification separately.