The Kafka Event Platform
We were losing audit data one HTTP call at a time — silently, and worst exactly when we were busiest. How I moved the platform onto a durable event backbone without a single flag-day cutover.
Every change to an appointment is worth remembering — a reschedule, a status change, a room move, a cancellation. Each is a log entry, and those entries are how the product answers "what happened to this booking, and when?"
The volume is the story. North of 3,500 active clinics; for each, an automated hook reconciles appointments every few minutes; a single request can carry a couple hundred appointments, each generating a log entry. Multiply it out and you get a firehose of small writes that never stops.
Silent data loss is the failure
that never pages you.
Each log entry was written with its own synchronous HTTP call to a dedicated logging service. Fine in a code review; fine at low volume. But one reconciliation request carrying 200 appointments fanned out into 200 blocking calls before it could finish — times thousands of clinics, on a few-minute cadence.
Under that load the logging service fell over. And a synchronous call to a service that isn't answering doesn't queue anything — it fails, and the data evaporates. No buffer, no retry, no durable place to wait. The loss clustered exactly when volume was highest.
This wasn't a flame-graph hunt. The failure mode was understood; the work was confirming shape and scale before proposing a rewrite. I traced the call path and confirmed two things.
One: the write really was per-record and synchronous, on the request path. Two — a gift — the exact same log payload was already used in several places across the platform.
That second fact shaped the entire migration. The measurement that mattered wasn't a profiler — it was understanding the failure honestly: what exactly is lost, when, and why.
Which options you reject says more than the one you pick.
Scale the logging service
RejectedTreats the symptom. The coupling remains — a degraded store still back-pressures the business path. You can't scale out of "the fast path is chained to the slow path."
Fire-and-forget async in-process
RejectedFixes latency, worsens loss — failures become fully invisible, still no durable buffer. Speed without durability is the wrong trade for audit data.
Persistent outbox, exactly-once
Rejected (for now)The most robust on paper — and I took it seriously. But it couples logging back into the business store's transactions, and the guarantee it buys was stronger than this data needs. I noted exactly where I'd reach for it if requirements changed.
Durable event log + dedicated async consumer
ChosenPublish each event to a durable, replayable log; a separate service consumes and persists asynchronously. The log absorbs bursts, survives a down consumer, and scales independently.
Produce to a durable log, consume asynchronously, persist idempotently. Three moves, each decoupling the fast work from the durable work.
Producer: publish, don't call
Services publish the log event to a durable stream instead of a cross-service HTTP round-trip — wrapped in a bounded in-memory buffer, exponential backoff, auto-reconnect, and a graceful drain on shutdown.
Consumer: a service built from scratch
A dedicated NestJS service whose only job is to consume events and persist them — isolated failures, consumer-group parallelism so throughput grows by adding consumers.
Sink: idempotent, unique-keyed writes
Every event carries a deterministic key; the consumer upserts on it. Redelivery upserts the same document — no duplicates, by construction.
The durable log sits between the fast work and the slow work.
Producers publish and move on. The topic absorbs bursts and survives a down consumer. The consumer group persists on its own schedule, idempotently.
I did not build exactly-once. I built at-least-once made safe by idempotent writes — the pragmatic sweet spot, without the cost of guaranteeing uniqueness end-to-end.
append-only, immutable records with unique identities → at-least-once is safe by construction
Exactly-once wasn't the answer.
Idempotency was.
The detail I'm proudest of is what made it safe to ship: the migration was drop-in because I kept the payload identical. The new event carried the exact same shape as the old HTTP body — so switching a service from "call" to "publish" was a small, reversible change with no downstream schema impact.
// old — synchronous, dropped data under load logChange(entry) → HTTP POST logging service // new — durable, drop-in (same payload) logChange(entry) → publish log-events topic
Services were moved over gradually, verified, and only then was the old path retired. No big-bang, no flag day — you make the two worlds coexist and walk the traffic across.
Eventual consistency
Bounded in-memory buffer
A second service + topic
There's a failed-events topic too — honestly, a safety net created during testing that production hardening made unnecessary, not an actively-drained DLQ. I'd rather say that than dress it up.
Silent data-loss events. The business path no longer depends on the logging store's health; the topic absorbs bursts.
eliminatedMessages retained across a well-partitioned, replicated topic — an eight-figure-message backbone where before we had a call that dropped data.
prod dashboardActive clinics whose every appointment change now lands durably, instead of dropping under load.
prod dashboardThe honest part: under peak load the consumer doesn't fully keep pace — meaningful lag, traced to one-at-a-time upserts and running fewer consumers than partitions. We scaled consumers up once as a first mitigation; the real remediation — batched writes + scaling toward the partition count — is in progress. A known, diagnosed, addressable bottleneck, not a mystery.
The core design I'd keep. Three things operating it taught me to do differently.
Batch writes from day one
The single biggest cause of consumer lag is one record per round-trip. Accumulate N, write in one bulk op — obvious in hindsight, and what's being built now.
Match consumers to partitions by default
Under-utilizing partitions leaves free throughput on the table. Make parallelism a deployment default so the pipeline scales without someone bumping a number.
Lag + throughput dashboards before shipping
The lag was found by looking, not by an alert — that ordering is backwards. Observability belongs in the definition of done, so the pipeline tells you when it's falling behind.