The Google File System

The problem: when something is always broken

File systems of 2003 assumed that disks mostly worked, that files were modest, and that programs read and wrote them in small pieces at random. Google’s machines broke every one of those assumptions. It ran thousands of cheap PCs with commodity disks, so at any moment some were dead and some were quietly corrupting data. Its files were web crawls measured in gigabytes and terabytes, written once by a stream of appends and read in long sequential sweeps by programs like MapReduce. A file system built for a workstation, scaled up, would spend its life fighting the hardware and the workload at once.

So the authors wrote down their assumptions first, which is why the paper is still worth reading as a method even where its design has been superseded: failure is normal; files are huge; appends dominate; bandwidth matters more than latency; and the file system and the applications are built by the same company, so the interface can be bent to fit.

The idea: one master for the map, many servers for the bytes

A GFS cluster has a single master and many chunkservers. Files are cut into fixed 64-megabyte chunks, each stored on three chunkservers. The master keeps all metadata in memory: the file namespace, which chunks make up each file, and where every chunk’s replicas are. Clients ask the master where a chunk lives, cache the answer, and then read or write the bytes directly from chunkservers. The master handles a few bytes per operation; the terabytes never pass through it.

How it works

The master expects to lose chunkservers and treats a missing replica as ordinary work. It exchanges heartbeats with every server, and when one goes silent the chunks it held become under-replicated and are cloned elsewhere, most urgent first; when the server returns, its stale copies are garbage-collected lazily. Chunkservers checksum every 64-kilobyte block, so a disk that lies is caught before its data spreads. The master’s own state is protected by an operation log replicated to other machines and by periodic checkpoints; if it dies, a new master replays the log and asks the chunkservers what they hold.

Writes separate data from control. The client pushes the bytes along a chain of replicas, each forwarding to the next nearest, so every network link is used once. Then it asks the primary replica, which holds a short lease from the master, to apply the write; the primary chooses a serial order and tells the others to apply it in that order. One machine decides, briefly and locally, and no global coordination is needed.

The interface is bent in one famous place. Record append lets many clients append to one file concurrently with GFS choosing the offset, guaranteeing that each record lands at least once, somewhere, atomically, possibly with duplicates or padding between records. That is a weaker promise than a POSIX file makes and a much stronger one than Google’s producer-consumer pipelines needed; applications tolerated duplicates with sequence numbers and checksums, and the file system stayed fast.

What it changed

GFS was the floor under MapReduce and Bigtable, and its open-source copy, the Hadoop Distributed File System, was the floor under the big-data industry: the NameNode and DataNodes are the master and chunkservers under other names. Inside Google it was replaced by Colossus around 2010, which spread the metadata across many machines and used erasure coding instead of three copies, because the single master finally did become the limit. What survived is the posture: design for failure as the normal case, replicate at the file-system layer so applications need not, and keep the metadata path and the data path apart.

Where you meet it today

• HDFS and every Hadoop-era cluster; the NameNode is still the single master.
• Object stores. S3 and its peers keep the separation of a metadata service from storage nodes, with erasure coding where GFS used replication.
• Chunking. Splitting large objects into fixed-size, independently replicated pieces is now how backups, video and ML datasets are stored.
• Leases. A short-lived grant of authority from a coordinator, renewed by heartbeat, is the standard way to pick a primary without consensus on every write.