Background work: queues, schedulers and cron at scale
Not everything happens in a request; the work that happens later needs the same design care, and usually gets none.
Every request handler eventually accumulates barnacles — send the email, resize the image, sync the CRM — until p99 is a tour of your integrations. The fix is one honest sentence: acknowledge now, do it later. But “later” is a system, and because nobody designs it on purpose, it is where reliability quietly goes to die. This chapter designs it on purpose.
A task queue is not an event stream
The streams chapter’s log broadcasts facts to whoever cares, retained and replayable. A task queue distributes work: each task goes to exactly one worker, gets acknowledged on success, is redelivered on timeout, and after N failures lands in a dead-letter queue — parked with its error for a human, because infinite retries on a poison task is a treadmill, not persistence. Choose by noun: fact → log; chore → queue. (Kafka can back a work queue; you will simply be implementing redelivery and DLQs yourself — sometimes worth it, never free.)
Workers live under Distributed Systems law
Delivery is at-least-once — the worker that finishes and dies before acking means the task will run twice — so every handler is idempotent or it is wrong: charge-payment carries an idempotency key; send-email checks a sent-record; resize-image overwrites the same derivative key (naturally idempotent — the best kind). Two operational patterns complete the worker: a visibility timeout comfortably above real task duration (too short is a duplicate-work generator), and per-tenant fairness — one customer’s 3-million-task import must not starve everyone’s password resets, so partition queues by priority and/or tenant and cap per-tenant concurrency. Priority inversion is real: the bulk lane never blocks the urgent lane.
Later, and repeatedly: timers and distributed cron
“Send the reminder in 24 hours” wants delayed delivery (native in good queues; otherwise a sorted-by-due-time store swept by a poller — a timer wheel at heart). “Every night at 2:00” wants cron — and distributed cron is a trap wearing a crontab: run the schedule on three boxes for availability and the job fires three times. The correct shape is this chapter’s SRE reading: leader-elect the scheduler (coordination chapter — lease plus fencing token), the leader enqueues tasks rather than doing work, and fired-at state is recorded so a failover neither skips nor repeats the 2:00 run. Idempotent jobs make even that bookkeeping forgiving. And respect the humble timezone: “9 a.m. local, per user” is a per-timezone fan-out with DST edges — schedule in UTC, map at enqueue time.
Observability for the invisible
Background work fails silently by default — nobody is waiting on the response. The dashboard that keeps it honest: queue depth and — better — age of the oldest task (depth says busy; age says late), per-queue failure and retry rates (a retry surge is an incident with a fuse), DLQ size with an alert on first arrival, and worker saturation for the capacity story. Little’s Law closes the loop: arrival rate × average task time = workers needed; when age climbs, that equation names the deficit.