Embeddings, Explained: From a Word Corpus to Vectors That Mean Something

Every RAG pipeline, every semantic search feature, every "find similar documents" tool starts with the same question: how do you turn text into numbers a model can reason about? That's what an embedding is — a vector that represents a word, sentence, or document such that similar meaning ends up close together in vector space. This post builds that idea up from scratch: a raw sentence, a tokenized corpus, and then every major embedding technique on top of it.

1. Start with a corpus

A corpus is just your collection of text. Before you can embed anything, you need to tokenize it — split raw text into words (tokens) — and usually clean it up a bit. NLTK is the standard library for this in Python.

pip install nltk

import nltk
nltk.download("punkt_tab")
nltk.download("stopwords")
 
from nltk.tokenize import word_tokenize
from nltk.corpus import stopwords
 
corpus = [
    "PyTorch makes it easy to build neural networks.",
    "Embeddings turn words into vectors that capture meaning.",
    "Neural networks learn patterns from data.",
]
 
stop_words = set(stopwords.words("english"))
 
tokenized = [
    [w.lower() for w in word_tokenize(sentence) if w.isalpha() and w.lower() not in stop_words]
    for sentence in corpus
]
 
print(tokenized)
# [['pytorch', 'makes', 'easy', 'build', 'neural', 'networks'],
#  ['embeddings', 'turn', 'words', 'vectors', 'capture', 'meaning'],
#  ['neural', 'networks', 'learn', 'patterns', 'data']]

That list of token lists is your corpus, cleaned and ready. Every embedding method below starts from something like this.

NLTK docs: nltk.org

2. The simplest representation: one-hot encoding

Give every unique word in your vocabulary its own index, and represent each word as a vector of zeros with a single 1 at that index.

vocab = sorted(set(w for sentence in tokenized for w in sentence))
word_to_idx = {w: i for i, w in enumerate(vocab)}
 
def one_hot(word):
    vec = [0] * len(vocab)
    vec[word_to_idx[word]] = 1
    return vec

This works, but it has two real problems: the vectors are huge and sparse (one dimension per vocabulary word), and every word is equally "far" from every other word — "neural" and "networks" look just as unrelated as "neural" and "pytorch". There's no notion of meaning here at all, just identity.

3. Count-based embeddings: Bag of Words and TF-IDF

Bag of Words (BoW) represents a sentence as a vector of word counts. TF-IDF (Term Frequency–Inverse Document Frequency) improves on that by down-weighting words that appear in almost every document (like "the") and up-weighting words that are distinctive to a particular document.

from sklearn.feature_extraction.text import TfidfVectorizer
 
sentences = [" ".join(tokens) for tokens in tokenized]
vectorizer = TfidfVectorizer()
tfidf_matrix = vectorizer.fit_transform(sentences)
 
print(vectorizer.get_feature_names_out())
print(tfidf_matrix.toarray())

Still no real notion of meaning — "good" and "great" are just as unrelated as "good" and "car" — but it's a solid, cheap baseline that's still used for search and document ranking today.

4. Word2Vec: the first embeddings that captured meaning

Word2Vec (Mikolov et al., 2013) was the breakthrough that made "embeddings" mean something specific: dense, low-dimensional vectors (typically 100–300 dimensions) trained so that words appearing in similar contexts end up with similar vectors. It has two training approaches:

• CBOW (Continuous Bag of Words) — predict a word from its surrounding context words.
• Skip-gram — predict the surrounding context words from a single word. Slower to train, generally better on rare words.

pip install gensim

from gensim.models import Word2Vec
 
model = Word2Vec(
    sentences=tokenized,
    vector_size=100,   # embedding dimensions
    window=5,           # context window size
    min_count=1,        # ignore words below this frequency
    sg=1,                # 1 = skip-gram, 0 = CBOW
)
 
vector = model.wv["neural"]              # a 100-dim vector
similar = model.wv.most_similar("neural") # closest words by cosine similarity

This is where the famous king - man + woman ≈ queen result comes from — the vector arithmetic works because the directions in the embedding space encode relationships, not just identity.

Gensim docs: radimrehurek.com/gensim · Original paper: arxiv.org/abs/1301.3781

5. GloVe and fastText: two variations on the idea

• GloVe (Stanford, 2014) trains on global word co-occurrence statistics across the whole corpus rather than a sliding local window, which tends to capture broader corpus-level patterns.
• fastText (Meta, 2016) embeds subword character n-grams, not whole words — so it can produce a reasonable vector for a misspelled or unseen word by composing it from familiar pieces.

import gensim.downloader as api
glove = api.load("glove-wiki-gigaword-100")
glove.most_similar("neural")

GloVe: nlp.stanford.edu/projects/glove · fastText: fasttext.cc

6. The limitation all of these share: one vector per word

Word2Vec, GloVe, and fastText all give a word exactly one vector, no matter the context. "bank" gets the same embedding in "river bank" and "savings bank." That's the ceiling static embeddings hit — and the problem contextual embeddings were built to solve.

7. Contextual embeddings: a different vector every time

Starting with ELMo (2018) and then BERT (2018) and its successors, embeddings stopped being a fixed lookup table and became the output of a neural network run on the whole sentence — so the vector for a word depends on what's around it.

from transformers import AutoTokenizer, AutoModel
import torch
 
tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
model = AutoModel.from_pretrained("bert-base-uncased")
 
inputs = tokenizer("The bank raised interest rates.", return_tensors="pt")
with torch.no_grad():
    outputs = model(**inputs)
 
# one contextual vector per token, shape: (1, seq_len, 768)
token_embeddings = outputs.last_hidden_state

Run the same code with "He sat on the river bank." and the vector for "bank" comes out different — that's the whole point.

BERT paper: arxiv.org/abs/1810.04805 · Hugging Face docs: huggingface.co/docs/transformers

8. Sentence and document embeddings

For search, RAG, and clustering, you usually want one vector per sentence or chunk, not per token. Two common approaches:

• Mean pooling — average the contextual token embeddings from a model like BERT.
• Sentence-Transformers — models fine-tuned specifically so that cosine similarity between sentence vectors reflects actual semantic similarity, which raw BERT embeddings aren't reliably good at out of the box.

from sentence_transformers import SentenceTransformer
 
model = SentenceTransformer("all-MiniLM-L6-v2")
embeddings = model.encode([
    "PyTorch makes it easy to build neural networks.",
    "Neural nets are simple to construct with PyTorch.",
    "The stock market fell sharply today.",
])
 
from sentence_transformers.util import cos_sim
print(cos_sim(embeddings[0], embeddings[1]))  # high similarity
print(cos_sim(embeddings[0], embeddings[2]))  # low similarity

Sentence-Transformers docs: sbert.net

9. Where this fits into RAG and agents

This is the same building block behind retrieval-augmented generation and agent memory: embed your documents once, store the vectors in a vector database (FAISS, Pinecone, pgvector), then embed a query at runtime and retrieve the nearest stored vectors by cosine similarity. Everything above — from TF-IDF to sentence-transformers — is a candidate for that embedding step; which one you pick is a tradeoff between cost, latency, and how much semantic nuance you actually need.

Reference links

• NLTK: nltk.org
• Gensim (Word2Vec, GloVe loading): radimrehurek.com/gensim
• Word2Vec paper: arxiv.org/abs/1301.3781
• GloVe: nlp.stanford.edu/projects/glove
• fastText: fasttext.cc
• BERT paper: arxiv.org/abs/1810.04805
• Hugging Face Transformers: huggingface.co/docs/transformers
• Sentence-Transformers: sbert.net

Next in this series: building a small RAG pipeline on top of these embeddings, from chunking to retrieval.