Efficient Estimation of Word Representations in Vector Space

The problem: to a computer, every word was equally unlike every other

The standard way to hand a word to a learning algorithm in 2012 was a one-hot vector: a list of fifty thousand zeros with a single one at the position for that word. “King” and “queen” shared nothing in that representation; neither did “walk” and “walked”. Whatever a model learned about one word taught it nothing about any other, and the long tail of rare words was hopeless. Richer representations existed, learned as a by-product of neural language models, but training them took weeks and they did not scale to the vocabularies and corpora that mattered.

The idea: you shall know a word by the company it keeps, at scale

Mikolov and colleagues proposed two deliberately simple models. Skip-gram takes a word and tries to predict the words around it; continuous bag of words does the reverse, predicting a word from its neighbours. Neither has a hidden layer in the usual sense: a word is looked up in a table of vectors, and that vector is used directly to score the predictions. The vectors start random and are nudged, billions of times, towards making the predictions right. The cheapness is the point; it let them train on 1.6 billion words in under a day, where earlier models had taken weeks on far less.

How it works

The result the paper is remembered for was not planned. Because the model is linear, the vectors capture relationships as consistent directions, and arithmetic on them works: subtract “man” from “king”, add “woman”, and the nearest vector is “queen”. The same holds for capitals and countries, tenses, plurals and comparatives. The authors built a test set of nearly twenty thousand such analogy questions to measure it. The simulation below shows the geometry in two dimensions; the real vectors have three hundred.

The companion paper later that year supplied the engineering that made it practical: negative sampling, which replaces the expensive prediction over the whole vocabulary with a cheap contrast against a few random words, and subsampling of frequent words like “the” that carry little signal. Together they are what the open-source word2vec tool shipped, and what most people mean by the name.

What it changed

Embeddings became the standard first layer of every language model and then of models for everything else: products (item2vec), users, songs, proteins, graph nodes. Stanford’s GloVe and Facebook’s fastText refined the idea; contextual embeddings from ELMo and BERT in 2018 made the vector depend on the sentence, which fixed word2vec’s inability to tell a river bank from a savings bank. The approximate-nearest-neighbour search that embeddings demand grew into today’s vector databases, and retrieval-augmented generation is, at bottom, an embedding lookup.

Where you meet it today

• Semantic search and recommendation. “Similar items” and “you might also like” are nearest-neighbour queries over embeddings.
• Vector databases and RAG. Pinecone, pgvector and friends index the descendants of these vectors; a RAG pipeline embeds the query and finds its neighbours.
• The first layer of every LLM is still a lookup table of token vectors, learned the same way, just larger.
• Bias audits. The analogy trick also surfaced stereotypes in training text, which began the study of bias in learned representations.