Derived data: search, analytics and event sourcing

By this point your system has one source of truth and a growing crowd of shadows: a cache, a search index, an analytics store, maybe a recommendation feed. The liberating reframe of this chapter: those are all the same thing — derived data, projections rebuilt from the source — and the log you built last chapter is how they stay honest.

The derived-data mindset

A derived store is disposable by definition: if it can be rebuilt from the source of truth, it needs no backups, no two-phase writes, no heroics — it needs a pipeline (consume the change stream, transform, upsert) and a freshness SLO (“search reflects writes within 30 seconds”), monitored as consumer lag. Rebuild-from-scratch is not an emergency procedure; it is the design test. If you cannot replay the log into an empty index and get the right answer, the pipeline is wrong now — you just have not noticed yet.

Search — a different index for a different question

B-trees answer “rows where author = X”; search answers “documents about X” — and needs the inverted index: for every term, the list of documents containing it. Queries intersect those lists, then rank (BM25 remains the honest baseline: rare terms count more, term-stuffed long docs count less). The craft hides in analysis — how text becomes terms: lowercase, tokenise, stem (“running” → “run”), handle synonyms — and analysis mismatches between index-time and query-time explain most “search is broken” tickets. Populate it like every derived store: CDC from the source, never dual writes from application code (the streams chapter already prosecuted that crime).

OLTP and OLAP — stop making one store do both

Transactional stores read rows; analytics reads columns — “average order value by month for two years” touches two columns of fifty million rows, and a row store drags the other forty-eight through memory (the ladder, wasted). Column stores lay data column-wise: scan only what you select, compress beautifully (a column is self-similar), aggregate at vector speed. So the standard shape: OLTP databases for the product, a warehouse/lakehouse for questions, the log in between — batch loads where hours of lag are fine, streaming where minutes matter, and reprocessing as a first-class operation either way, because the transform logic will change and history must be re-derivable.

Event sourcing and CQRS — the mindset at its limit

Event sourcing promotes the log to be the source of truth: state is a projection, every past state reconstructable, audit for free. CQRS is its natural partner — commands append events, queries hit projections shaped per reader. The unbought costs: eventual consistency between write and read sides (session guarantees return from the Distributed Systems pathway), and event schema evolution — your events are forever, so the RPC chapter’s compatibility rules apply to your own history. Superb for domains that are ledgers (payments, inventory, anything audited); a self-inflicted distributed system for a CRUD app. Most systems want the mindset — log-fed, rebuildable projections — without the full commitment.