Introduction & Text Preprocessing

What Is Natural Language Processing?

Natural Language Processing (NLP) is the subfield of AI concerned with enabling computers to understand, interpret, generate, and manipulate human language.

The Scope of NLP

NLP spans a vast range of tasks:

Category	Tasks
Understanding	Sentiment analysis, NER, parsing, QA
Generation	Machine translation, summarization, chatbots
Retrieval	Search, semantic similarity, document clustering
Speech	Speech-to-text (ASR), text-to-speech (TTS)

Why Is NLP Hard?

Language is deeply ambiguous and context-dependent:

• Lexical ambiguity: "bank" = financial institution OR river bank
• Syntactic ambiguity: "I saw the man with the telescope" — who has the telescope?
• Semantic ambiguity: "The chicken is ready to eat" — hungry chicken or food?
• Pragmatic ambiguity: "Can you pass the salt?" — capability question or polite request?
• World knowledge: "The trophy didn't fit in the suitcase because it was too big" — what is "it"?

The Classic NLP Pipeline

Raw Text
  ↓
[1] Text Preprocessing  (tokenization, normalization, cleaning)
  ↓
[2] Linguistic Analysis (morphology, syntax, semantics)
  ↓
[3] Representation      (BoW, TF-IDF, embeddings)
  ↓
[4] Modeling            (classification, generation, extraction)
  ↓
[5] Evaluation & Deployment

Tokenization

Tokenization splits raw text into discrete units (tokens) — typically words, subwords, or characters.

Types of Tokenization

Type	Example Input → Output	Use Case
Word tokenization	"don't" → ["don't"] or ["do", "n't"]	Classic NLP tasks
Sentence tokenization	Paragraph → list of sentences	Summarization, parsing
Subword tokenization	"unhappiness" → ["un", "happy", "ness"]	Transformers (BERT, GPT)
Character tokenization	"cat" → ["c", "a", "t"]	Very small vocabularies, some seq2seq

Subword Tokenization Algorithms

Modern LLMs use subword tokenization to handle unknown words and keep vocabulary size manageable:

• BPE (Byte-Pair Encoding): merges the most frequent byte/character pairs iteratively — used by GPT-2, RoBERTa
• WordPiece: similar to BPE but uses likelihood instead of frequency — used by BERT
• SentencePiece: language-agnostic; treats the text as a raw byte sequence — used by T5, LLaMA

Challenges in Tokenization

• Contractions: "I'm" → "I" + "am" or "I" + "'m"?
• Hyphenation: "state-of-the-art" → 1 token or 4?
• URLs/emails: should not be split
• Languages without spaces: Chinese, Japanese, Thai need different tokenizers

Text Normalization

Normalization standardizes text to reduce surface-level variation without losing meaning.

Common Normalization Steps

1. Lowercasing: "NLP" → "nlp" (careful: "US" ≠ "us" always)
2. Punctuation removal: strip or replace with spaces
3. Number handling: "100" → "NUM" or keep as-is
4. Contraction expansion: "won't" → "will not"
5. Unicode normalization: "café" → "cafe" (NFD/NFC)
6. Noise removal: HTML tags, URLs, special characters, emojis

Stopword Removal

Stopwords are high-frequency words (articles, prepositions, conjunctions) with little semantic content: "the", "is", "at", "which", "on".

• Removing them reduces feature space and noise for bag-of-words models
• Do NOT remove stopwords when order or grammar matters (parsing, translation, question answering)

Stemming vs Lemmatization

Both reduce words to a base form to consolidate variants, but they work differently.

Stemming

Stemming applies rule-based heuristics to chop off word suffixes:

$\text{running} \xrightarrow{\text{stem}} \text{run} \qquad \text{studies} \xrightarrow{\text{stem}} \text{studi}$

• Fast and simple
• Output may not be a real word ("studi", "happi")
• Common algorithms: Porter Stemmer, Snowball (Porter2), Lancaster

Lemmatization

Lemmatization uses vocabulary and morphological analysis to return the dictionary form (lemma):

$\text{running} \xrightarrow{\text{lemma}} \text{run} \qquad \text{studies} \xrightarrow{\text{lemma}} \text{study} \qquad \text{better} \xrightarrow{\text{lemma}} \text{good}$

• Slower — needs POS context ("better" → "good" only as adjective)
• Always returns a real word
• More accurate; needed when word meaning matters

When to Use Which

Criterion	Stemming	Lemmatization
Speed requirement	✓ Fast	✗ Slower
Semantic accuracy	✗ Rough	✓ Precise
Use case	Information retrieval, search	QA, sentiment analysis, parsing
Needs POS info	No	Yes

import re
import string
import nltk
import spacy
from nltk.tokenize import word_tokenize, sent_tokenize
from nltk.corpus import stopwords
from nltk.stem import PorterStemmer, SnowballStemmer
from nltk.stem import WordNetLemmatizer

nltk.download(['punkt', 'stopwords', 'wordnet', 'averaged_perceptron_tagger'], quiet=True)

# ─── Sample text ─────────────────────────────────────────────
text = """Natural Language Processing (NLP) is AMAZING! It's transforming how
computers understand human language. Running, runs, and ran are all forms of 'run'.
Visit https://example.com for more info. <b>HTML tags</b> should be removed."""

# ─── 1. Basic Cleaning ────────────────────────────────────────
def clean_text(text: str) -> str:
    text = re.sub(r'<[^>]+>', '', text)          # remove HTML tags
    text = re.sub(r'https?://S+', '', text)      # remove URLs
    text = re.sub(r's+', ' ', text).strip()      # collapse whitespace
    return text

# ─── 2. Normalization ─────────────────────────────────────────
contractions = {"it's": "it is", "won't": "will not", "can't": "cannot",
                "i'm": "i am", "they're": "they are", "don't": "do not"}

def expand_contractions(text: str) -> str:
    for contraction, expansion in contractions.items():
        text = re.sub(contraction, expansion, text, flags=re.IGNORECASE)
    return text

# ─── 3. Tokenization ─────────────────────────────────────────
cleaned = clean_text(text)
expanded = expand_contractions(cleaned)
lowered = expanded.lower()

tokens = word_tokenize(lowered)
sentences = sent_tokenize(cleaned)
print(f"Word tokens ({len(tokens)}): {tokens[:10]}")
print(f"Sentences ({len(sentences)}): {sentences[:2]}")

# ─── 4. Stopword and Punctuation Removal ─────────────────────
stop_words = set(stopwords.words('english'))
tokens_clean = [t for t in tokens if t not in stop_words and t not in string.punctuation]
print(f"After stopword removal ({len(tokens_clean)}): {tokens_clean[:10]}")

# ─── 5. Stemming ─────────────────────────────────────────────
porter = PorterStemmer()
snowball = SnowballStemmer('english')
words_demo = ['running', 'studies', 'happiness', 'beautiful', 'better']
print("\nStemming comparison:")
print(f"{'Word':<15} {'Porter':<15} {'Snowball':<15}")
for w in words_demo:
    print(f"{w:<15} {porter.stem(w):<15} {snowball.stem(w):<15}")

# ─── 6. Lemmatization ────────────────────────────────────────
lemmatizer = WordNetLemmatizer()
print("\nLemmatization (with POS):")
word_pos_pairs = [('running', 'v'), ('studies', 'v'), ('better', 'a'), ('geese', 'n')]
for word, pos in word_pos_pairs:
    print(f"  {word:<12} → {lemmatizer.lemmatize(word, pos=pos)}")

# ─── 7. spaCy Lemmatization (preferred — uses context) ───────
nlp_spacy = spacy.load("en_core_web_sm")
doc = nlp_spacy("The geese are running quickly. Studies show better results.")
print("\nspaCy Lemmatization:")
for token in doc:
    if not token.is_stop and not token.is_punct:
        print(f"  {token.text:<15} → lemma: {token.lemma_:<15} pos: {token.pos_}")

# ─── 8. Subword Tokenization (Hugging Face) ──────────────────
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
bert_tokens = tokenizer.tokenize("unhappiness and misunderstanding")
print(f"\nBERT subword tokens: {bert_tokens}")
# → ['un', '##happiness', 'and', 'mis', '##understanding']

Summary

Text preprocessing pipeline:

1. Clean: remove HTML, URLs, noise
2. Normalize: expand contractions, unicode
3. Lowercase (context-dependent)
4. Tokenize: word/sentence/subword depending on model
5. Remove stopwords (only for BoW-based tasks)
6. Stem or lemmatize (traditional ML only)

Key decisions:

• Use lemmatization over stemming when word meaning matters
• Use subword tokenization (BPE/WordPiece) for transformer models
• Match preprocessing to the task — no one-size-fits-all pipeline

Next: Phonology & Speech Processing — how computers interpret spoken language.