Large-scale cluster management at Google with Borg

The problem: a hundred thousand machines and no idle ones allowed

Every Google service, from search to Gmail, and every batch job, from MapReduce to the nightly index build, had to run somewhere. Giving each team its own machines was simple and wasteful: serving systems need headroom for their peaks, batch jobs need bursts, and dedicated clusters sit half-empty most of the day. The alternative, one shared pool for everything, raises the questions that make cluster management hard. Who goes first when the pool is full? What happens to a batch job when a serving job needs its machine? How does a service find its instances when they can be anywhere? And how do you keep all of it running while machines, racks and the management system itself fail?

Google answered those questions internally around 2004 and ran essentially everything on the result for a decade before describing it. The paper is the description, and it was published a year after the same team released Kubernetes, which is the public retelling of its lessons.

The idea: one scheduler, priorities, and the slack is for batch

Borg manages a cell, a cluster of about ten thousand machines, as a single pool. Users submit jobs made of identical tasks, each with a priority and a resource request. The scheduler finds machines with room and places tasks; a Borglet agent on each machine starts them in containers and reports back. Every job carries a priority, and the bands matter: production jobs are placed as though batch jobs did not exist, and a batch task simply gets evicted when a production task needs its machine, to be rescheduled later. Batch work fills the gaps that production headroom leaves and gives them back on demand.

How it works

The Borgmaster is a replicated state machine (the Raft-style idea, with Paxos) holding every job, task and machine; it survives the loss of most of its replicas, and even a complete outage leaves running tasks running, because the Borglets keep going without it. The scheduler scans the queue in priority order, finds feasible machines, and scores them so that tasks spread for failure tolerance while leaving large holes for large tasks. Tasks get stable names through a service that lets clients find instances wherever they land, and every task exports metrics and health that the system scrapes.

Two refinements made the economics work. Tasks ask for more resources than they use, so Borg measures actual use and reclaims the difference for lower-priority work, giving it back if the owner’s usage rises. And the system was shaped by the people running it: a built-in web interface showed every job’s state and every machine’s load, and the paper is frank that the tooling was as important as the scheduler.

What it changed

Borg’s lessons, good and bad, became Kubernetes. Pods replaced bare tasks, because co-scheduled helper processes were a pattern everyone had reinvented; labels replaced rigid job membership; one IP per pod replaced Borg’s awkward port juggling; a declarative API server replaced the Borgmaster’s remote procedure calls. The companion essay in ACM Queue, by the same authors, walks through those choices. Borg also normalised containers as the unit of deployment years before Docker, and made “bin-packing with priorities” the default model for every cluster scheduler since, including Mesos, YARN and Nomad.

Where you meet it today

• Kubernetes. Requests and limits, priority classes, preemption and node scoring are Borg’s mechanisms with Borg’s names barely changed.
• Spot and preemptible instances. The cloud sells the batch band to the public: cheap capacity that may be taken back.
• Resource reclamation. Vertical autoscalers and “right-sizing” tools do what Borg did automatically with over-requested resources.
• The control-plane pattern. A replicated store of desired state and agents that reconcile towards it is now how most infrastructure is built.