Streams and the log
Past a certain scale the question changes from “where is the data?” to “in what order did things happen?”
Sharding split the data; a second pressure arrives with it: every write starts owing side effects — update the search index, invalidate the cache, notify a service. Doing that synchronously makes users wait on your bookkeeping; doing it “later, somehow” loses updates on the first bad day. The instrument that tames it is old and perfect.
The log — one ordered history
For the side-effects problem, the instrument is old and perfect: the append-only log (Kafka being the industrial version). Producers append; the log assigns each record a position; consumers read at their own pace, remembering only an offset. Three properties fall out at once: producers and consumers are decoupled (a slow indexer delays nobody — its offset just lags, and that lag is your cleanest health metric); history is replayable (rewind to rebuild a cache or backfill a brand-new consumer from last month’s events); and order within a partition is settled once by position, never re-argued — the Distributed Systems pathway taught how precious that is. Partitions shard the log for throughput (same key → same partition → per-key order preserved); consumer groups spread partitions across workers, with the standing rule that parallelism ≤ partition count. Queues (SQS-style) differ from logs on one axis worth naming: a queue deletes on consumption and targets one handler per message; a log retains and lets many readers share one history.
Closing the last gap
“Commit the row, then publish the event” — a crash between the two either loses the event or invents one, and no care in ordering fixes it. Make one write the moment of truth: the transactional outbox writes the event into an outbox table inside the same database transaction as the row, and a relay ships outbox rows to the log afterwards (at-least-once, so consumers stay idempotent — permanently on duty). Or go one lower with change data capture: treat the database’s own replication log as the event stream, and derived systems — caches, indexes, warehouses — become consumers of the source of truth rather than hopeful recipients of app-code side effects. Event sourcing takes the idea to its limit (the log is the source of truth; state is a projection) — powerful for audit-heavy domains, a real modelling commitment everywhere else; know it exists, reach for it deliberately.
Long, and worth every paragraph. The log as the unifying abstraction behind replication, streams and integration.
The companion to this course — primary writing from the teams who ran these systems in production.