Text Representation & Embeddings

Why Representation Matters

ML models cannot process raw text — they need numerical representations. The choice of representation has enormous impact on model quality:

• Bag-of-Words: simple, interpretable, sparse, no word order
• TF-IDF: weighted BoW, better for search
• Word2Vec / GloVe: dense semantic vectors, captures word meaning
• FastText: handles out-of-vocabulary words via subwords
• BERT embeddings: contextual, state-of-the-art, handles polysemy

The Representation Learning Journey

Bag-of-Words  →  TF-IDF  →  Word2Vec/GloVe  →  ELMo  →  BERT/GPT
(1950s–1980s)   (1970s)     (2013–2014)       (2018)    (2018–now)
   sparse          sparse    dense static      context   deep context

Bag-of-Words (BoW) and TF-IDF

Bag-of-Words

BoW represents a document as a vector of word counts, ignoring order:

$\text{doc} = \text{"the cat sat on the mat"} \rightarrow \{\text{the}: 2, \text{cat}: 1, \text{sat}: 1, \text{on}: 1, \text{mat}: 1\}$

With vocabulary of size $V$, each document is a $V$-dimensional sparse vector.

Limitations:

• Ignores word order: "cat bites dog" = "dog bites cat"
• No semantics: "car" and "automobile" are unrelated vectors
• High dimensionality: typical vocabulary is 50,000–500,000 words

TF-IDF (Term Frequency-Inverse Document Frequency)

Downweights common words, upweights informative/rare ones:

$\text{TF-IDF}(t, d) = \underbrace{\frac{f_{t,d}}{\sum_{t'} f_{t',d}}}_{\text{TF}} \times \underbrace{\log\frac{N}{1 + |\{d: t \in d\}|}}_{\text{IDF}}$

• TF (Term Frequency): how often term $t$ appears in document $d$
• IDF (Inverse Document Frequency): log of (total docs / docs containing $t$); rare words get high IDF
• Common words ("the", "is") have low IDF → low TF-IDF weight

TF-IDF is still highly effective for text classification, information retrieval, and keyword extraction when combined with linear models.

Word2Vec

Word2Vec (Mikolov et al., 2013) trains word embeddings using a shallow neural network to predict:

• Skip-gram: given a center word, predict surrounding context words
• CBOW (Continuous Bag of Words): given context words, predict the center word

Skip-gram Objective

$\mathcal{L} = -\frac{1}{T}\sum_{t=1}^{T}\sum_{-c \leq j \leq c, j \neq 0} \log P(w_{t+j} | w_t)$

$P(w_o | w_i) = \frac{\exp(\mathbf{v}_{w_o}^T \mathbf{v}_{w_i})}{\sum_{w=1}^{W} \exp(\mathbf{v}_w^T \mathbf{v}_{w_i})}$

Negative sampling approximates the softmax by only updating the target word and $k$ random negative samples — makes training feasible.

The Word Analogy Property

The famous word arithmetic:

$\mathbf{v}_{\text{king}} - \mathbf{v}_{\text{man}} + \mathbf{v}_{\text{woman}} \approx \mathbf{v}_{\text{queen}}$

$\mathbf{v}_{\text{Paris}} - \mathbf{v}_{\text{France}} + \mathbf{v}_{\text{Germany}} \approx \mathbf{v}_{\text{Berlin}}$

This emerges from the training objective — it was not explicitly programmed.

GloVe (Global Vectors)

GloVe (Pennington et al., 2014) trains on global co-occurrence statistics:

$J = \sum_{i,j=1}^{V} f(X_{ij})(\mathbf{v}_i^T\tilde{\mathbf{v}}_j + b_i + \tilde{b}_j - \log X_{ij})^2$

where $X_{ij}$ = co-occurrence count of words $i$ and $j$. GloVe tends to produce slightly better embeddings than Word2Vec on analogy tasks.

FastText

FastText (Bojanowski et al., 2017) represents each word as a sum of subword character n-gram vectors:

$\mathbf{v}_{\text{apple}} = \mathbf{v}_{\langle app\rangle} + \mathbf{v}_{\langle ppl\rangle} + \mathbf{v}_{\langle ple\rangle} + \ldots$

Key advantage: handles out-of-vocabulary words — "grokking" can be represented even if never seen during training.

Static Embeddings Limitation: Polysemy

Word2Vec/GloVe assign one vector per word regardless of context:

• "I went to the bank" → same vector as "the river bank"
• BERT solves this with contextual embeddings.

Contextual Embeddings: BERT and Beyond

BERT (Bidirectional Encoder Representations from Transformers) generates a different embedding for each token depending on its full context:

• "I went to the bank to deposit money." → bank vector near financial institution
• "Fish live near the river bank." → bank vector near geography/nature

BERT is pre-trained with two tasks:

1. Masked Language Model (MLM): predict 15% randomly masked tokens
2. Next Sentence Prediction (NSP): classify if sentence B follows sentence A

Sentence Embeddings

Word-level BERT embeddings are not ideal for sentence-level tasks. Sentence-BERT (SBERT) fine-tunes BERT using siamese networks and a contrastive loss to produce semantically meaningful sentence vectors:

$\text{sim}(s_1, s_2) = \cos(\mathbf{e}_{s_1}, \mathbf{e}_{s_2})$

Popular sentence embedding models:

Model	Dim	Speed	Best For
all-MiniLM-L6-v2	384	Very fast	General semantic similarity
all-mpnet-base-v2	768	Moderate	Higher accuracy
text-embedding-ada-002 (OpenAI)	1536	API	Production, multilingual
E5-large	1024	Moderate	Retrieval, RAG
BGE-large	1024	Moderate	State-of-the-art retrieval

# ─── 1. TF-IDF ───────────────────────────────────────────────
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity
import numpy as np

corpus = [
    "Natural language processing is a subfield of artificial intelligence.",
    "Machine learning enables computers to learn from data.",
    "Deep learning uses neural networks with many layers.",
    "NLP tasks include sentiment analysis and machine translation.",
]
tfidf = TfidfVectorizer(max_features=100, stop_words='english')
X = tfidf.fit_transform(corpus)
print(f"TF-IDF matrix shape: {X.shape}")

# Top terms per document
feature_names = tfidf.get_feature_names_out()
for i, doc in enumerate(corpus):
    row = X[i].toarray()[0]
    top_idx = row.argsort()[-3:][::-1]
    top_terms = [(feature_names[j], round(row[j], 3)) for j in top_idx]
    print(f"  Doc {i+1} top terms: {top_terms}")

query = tfidf.transform(["sentiment analysis and NLP"])
sims = cosine_similarity(query, X).flatten()
print(f"\nQuery similarity scores: {sims.round(3)}")

# ─── 2. Word2Vec with Gensim ─────────────────────────────────
# pip install gensim
from gensim.models import Word2Vec, KeyedVectors
import gensim.downloader as api

# Train on toy corpus
sentences = [doc.lower().split() for doc in corpus * 50]  # replicate for demo
model = Word2Vec(sentences, vector_size=100, window=5, min_count=1,
                 workers=4, sg=1, epochs=50)  # sg=1: Skip-gram

# Most similar words
print("\nWord2Vec most similar to 'learning':")
try:
    similar = model.wv.most_similar("learning", topn=5)
    for word, score in similar:
        print(f"  {word:<20} {score:.3f}")
except KeyError:
    pass

# Load pretrained Google News Word2Vec (3M words, 300d)
# model_gn = api.load("word2vec-google-news-300")
# print("king - man + woman:", model_gn.most_similar(
#     positive=["king", "woman"], negative=["man"], topn=1))

# ─── 3. FastText ─────────────────────────────────────────────
from gensim.models import FastText

ft_model = FastText(sentences, vector_size=100, window=5, min_count=1,
                    workers=4, sg=1, epochs=50)

# Handle OOV (out-of-vocabulary)
oov_word = "neurolinguistics"
print(f"\nFastText can embed OOV word '{oov_word}':")
print(f"  Vector norm: {np.linalg.norm(ft_model.wv[oov_word]):.3f}")

# ─── 4. BERT Contextual Embeddings ───────────────────────────
from transformers import BertTokenizer, BertModel
import torch

tokenizer = BertTokenizer.from_pretrained("bert-base-uncased")
bert = BertModel.from_pretrained("bert-base-uncased")
bert.eval()

def get_bert_embedding(text, target_word):
    inputs = tokenizer(text, return_tensors="pt")
    with torch.no_grad():
        outputs = bert(**inputs)
    tokens = tokenizer.convert_ids_to_tokens(inputs["input_ids"][0])
    hidden = outputs.last_hidden_state[0]  # (seq_len, 768)
    # Find target word index
    for i, tok in enumerate(tokens):
        if target_word in tok:
            return hidden[i].numpy()
    return hidden.mean(0).numpy()

emb_bank_finance = get_bert_embedding("I went to the bank to deposit money.", "bank")
emb_bank_river   = get_bert_embedding("Fish swim near the river bank.", "bank")
sim = cosine_similarity([emb_bank_finance], [emb_bank_river])[0][0]
print(f"\nBERT: 'bank' (finance) vs 'bank' (river) cosine sim: {sim:.3f}")
# Should be <0.9 — BERT gives different embeddings for different senses

# ─── 5. Sentence Embeddings (SBERT) ──────────────────────────
from sentence_transformers import SentenceTransformer

sbert = SentenceTransformer("all-MiniLM-L6-v2")
sentences_eval = [
    "A man is playing guitar.",
    "A musician is strumming a guitar.",
    "The stock market fell 3% today.",
    "A puppy is chasing a ball in the park.",
]
embeddings = sbert.encode(sentences_eval, normalize_embeddings=True)
sim_matrix = embeddings @ embeddings.T
print("\nSentence similarity:")
for i in range(len(sentences_eval)):
    for j in range(i+1, len(sentences_eval)):
        print(f"  {sim_matrix[i,j]:.3f} | '{sentences_eval[i][:35]}' ↔ '{sentences_eval[j][:35]}'")

Summary

Method	Type	Handles OOV	Context-aware	Typical Dim
BoW	Sparse count	No	No	V (50k–500k)
TF-IDF	Sparse weighted	No	No	V (50k–500k)
Word2Vec	Dense static	No	No	100–300
GloVe	Dense static	No	No	50–300
FastText	Dense static + subword	Yes	No	100–300
ELMo	Dense contextual	No	Yes (LSTM)	1024
BERT	Dense contextual	Partial (WordPiece)	Yes (Transformer)	768
Sentence-BERT	Dense sentence	Partial	Yes	384–768

When to use what:

• Traditional ML + linear models: TF-IDF
• Word-level semantics, analogy: Word2Vec / GloVe
• Morphologically rich language or OOV: FastText
• Task-specific fine-tuning: BERT/RoBERTa
• Semantic similarity, search, RAG: Sentence-BERT / E5 / BGE

Next: Text Classification & Sentiment Analysis — using these representations for real tasks.