Text Classification & Sentiment Analysis

Text Classification

Text classification assigns a predefined label (or labels) to a piece of text. It underpins dozens of real-world applications:

Application	Labels
Spam detection	spam / ham
Sentiment analysis	positive / negative / neutral
Topic classification	news, sports, politics, tech, ...
Intent detection	book_flight, check_balance, cancel_order, ...
Language detection	en, fr, de, zh, ...
Toxicity detection	toxic / non-toxic

The Classification Pipeline

Raw Text
  ↓  Preprocessing (tokenize, normalize)
  ↓  Feature Extraction (TF-IDF, embeddings, BERT)
  ↓  Classifier (LR, SVM, BERT fine-tune)
  ↓  Label + Confidence Score

Naive Bayes Classifier

The classic text classifier applies Bayes' theorem with a conditional independence assumption:

$P(c | d) \propto P(c) \prod_{w \in d} P(w | c)$

• $P(c)$: prior probability of class $c$ (estimated from training data)
• $P(w | c)$: likelihood of word $w$ given class $c$
• Laplace smoothing prevents zero probabilities: $P(w|c) = \frac{\text{count}(w,c) + 1}{\sum_{w'} \text{count}(w',c) + |V|}$

MultinomialNB works well with TF-IDF features. Despite the naive independence assumption, it is competitive for spam detection and short text classification.

Logistic Regression and SVMs

Logistic Regression with TF-IDF features remains a strong baseline:

$P(y=1 | x) = \sigma(\mathbf{w}^T x + b) = \frac{1}{1 + e^{-(\mathbf{w}^T x + b)}}$

Trained with L2 regularization, it is often within 2–3% of fine-tuned BERT on many benchmarks while being 100× faster. LinearSVC frequently outperforms logistic regression on high-dimensional sparse TF-IDF features.

Sentiment Analysis

Levels of Granularity

Level	Input	Output	Example
Document-level	Full review	Positive/Negative/Neutral	★★★★☆ → Positive
Sentence-level	Single sentence	Polarity	"Great camera, but poor battery" → Mixed
Aspect-level (ABSA)	Sentence + aspects	Per-aspect polarity	camera=Pos, battery=Neg
Emotion detection	Text	Emotion category	"I can't believe they did that!" → Anger

Aspect-Based Sentiment Analysis (ABSA)

ABSA identifies both the aspect term (what is discussed) and aspect sentiment (how it is evaluated):

"The food was excellent but the service was slow."
  → (food, positive), (service, negative)

Modern approaches fine-tune BERT with specialized token tagging (B-ASP/I-ASP for aspect spans, then classify sentiment of each span).

Lexicon-Based Approaches

VADER (Valence Aware Dictionary and sEntiment Reasoner) uses a hand-crafted lexicon with rules for:

• Capitalization: "GREAT" > "great"
• Punctuation: "great!!!" > "great"
• Booster words: "very good" > "good"
• Negation: "not bad" → shifted score

Outputs a compound score in $[-1, +1]$. Excellent for social media text without fine-tuning.

Transfer Learning for Sentiment

Fine-tuned BERT-based models dominate benchmark leaderboards:

• bert-base-uncased fine-tuned on SST-2: ~93% accuracy
• RoBERTa-large fine-tuned on SST-2: ~96% accuracy
• Twitter-specific: cardiffnlp/twitter-roberta-base-sentiment
• Financial: ProsusAI/finbert
• Multilingual: nlptown/bert-base-multilingual-uncased-sentiment

Zero-Shot and Few-Shot Classification

Zero-shot classification uses an NLI (Natural Language Inference) model to classify without any task-specific fine-tuning:

$P(\text{label } l | \text{text}) \approx P_{\text{NLI}}(\text{entailment} | \text{text}, \text{"This text is about } l\text{"})$

Facebook's bart-large-mnli achieves competitive performance on many tasks without any labeled examples. Few-shot prompting of GPT-4 or Claude achieves near-SOTA on many classification tasks.

# ─── 1. TF-IDF + Naive Bayes / Logistic Regression ──────────
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.naive_bayes import MultinomialNB
from sklearn.linear_model import LogisticRegression
from sklearn.svm import LinearSVC
from sklearn.pipeline import Pipeline
from sklearn.metrics import classification_report, confusion_matrix
from sklearn.model_selection import train_test_split
import numpy as np

# Example: spam detection
texts = [
    "Win a free iPhone now! Click here!!!",
    "Hi, can we meet tomorrow for coffee?",
    "CONGRATULATIONS! You've won $1,000,000!",
    "Please find the attached report for Q3.",
    "Cheap meds online. No prescription needed.",
    "The meeting is scheduled for 3pm on Friday.",
    "You are selected for our exclusive offer!",
    "Thanks for your help with the project.",
]
labels = [1, 0, 1, 0, 1, 0, 1, 0]  # 1=spam, 0=ham

X_train, X_test, y_train, y_test = train_test_split(
    texts, labels, test_size=0.25, random_state=42)

for name, clf in [
    ("MultinomialNB", MultinomialNB()),
    ("LogisticRegression", LogisticRegression(max_iter=1000)),
    ("LinearSVC", LinearSVC()),
]:
    pipe = Pipeline([
        ("tfidf", TfidfVectorizer(ngram_range=(1,2), max_features=5000)),
        ("clf", clf),
    ])
    pipe.fit(X_train, y_train)
    preds = pipe.predict(X_test)
    print(f"\n{name}:")
    print(classification_report(y_test, preds, target_names=["ham","spam"]))

# ─── 2. VADER Sentiment (no training needed) ─────────────────
# pip install vaderSentiment
from vaderSentiment.vaderSentiment import SentimentIntensityAnalyzer

vader = SentimentIntensityAnalyzer()
reviews = [
    "This product is AMAZING! Best purchase ever!!!",
    "Not bad, but could be better.",
    "Terrible experience. Completely broken.",
    "The camera is great but the battery life is awful.",
]
print("\nVADER Sentiment Analysis:")
for text in reviews:
    scores = vader.polarity_scores(text)
    label = "Positive" if scores['compound'] > 0.05 else             "Negative" if scores['compound'] < -0.05 else "Neutral"
    print(f"  [{label:8}] {scores['compound']:+.3f} | {text[:50]}")

# ─── 3. HuggingFace Sentiment Pipeline ───────────────────────
from transformers import pipeline

# Domain-specific: financial sentiment
fin_sentiment = pipeline("text-classification",
                          model="ProsusAI/finbert")
financial_texts = [
    "Apple reported record quarterly earnings, beating all estimates.",
    "The company faces bankruptcy proceedings amid declining sales.",
    "Analysts maintain a neutral outlook pending further data.",
]
print("\nFinBERT Sentiment:")
for text in financial_texts:
    result = fin_sentiment(text)[0]
    print(f"  [{result['label']:8}] {result['score']:.3f} | {text[:60]}")

# ─── 4. Fine-tuning BERT for Classification ──────────────────
from transformers import AutoTokenizer, AutoModelForSequenceClassification
from transformers import TrainingArguments, Trainer
from datasets import Dataset
import torch

# Minimal fine-tuning example (SST-2 style)
model_name = "distilbert-base-uncased"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForSequenceClassification.from_pretrained(model_name, num_labels=2)

def tokenize(batch):
    return tokenizer(batch["text"], truncation=True, padding=True, max_length=128)

train_data = Dataset.from_dict({"text": texts, "label": labels})
train_data = train_data.map(tokenize, batched=True)
train_data.set_format("torch", columns=["input_ids", "attention_mask", "label"])

training_args = TrainingArguments(
    output_dir="./results",
    num_train_epochs=3,
    per_device_train_batch_size=8,
    logging_steps=10,
    save_strategy="no",
)
trainer = Trainer(model=model, args=training_args, train_dataset=train_data)
# trainer.train()  # uncomment to actually train

# ─── 5. Zero-Shot Classification ─────────────────────────────
zero_shot = pipeline("zero-shot-classification",
                     model="facebook/bart-large-mnli")
candidate_labels = ["technology", "politics", "sports", "entertainment", "finance"]

articles = [
    "The Fed raised interest rates by 25 basis points to combat inflation.",
    "Scientists unveiled a new quantum computing chip with 1000 qubits.",
    "The championship game went into overtime with a stunning comeback.",
]
print("\nZero-Shot Classification:")
for text in articles:
    result = zero_shot(text, candidate_labels)
    top_label = result["labels"][0]
    top_score = result["scores"][0]
    print(f"  [{top_label:15}] {top_score:.3f} | {text[:55]}")

Evaluation Metrics for Classification

Binary Classification

Metric	Formula	When to Use
Accuracy	$(TP+TN)/(TP+TN+FP+FN)$	Balanced classes
Precision	$TP/(TP+FP)$	Cost of false positives is high (spam)
Recall	$TP/(TP+FN)$	Cost of false negatives is high (cancer)
F1	$2 \cdot P \cdot R / (P+R)$	Imbalanced classes
AUC-ROC	Area under ROC curve	Ranking/threshold agnostic
PR-AUC	Area under PR curve	Heavily imbalanced datasets

Multi-Class Aggregation

• Macro-F1: average F1 per class, equal weight (favors minority classes)
• Weighted-F1: average F1 weighted by class support (standard reporting)
• Micro-F1: global TP/FP/FN (equals accuracy for multi-class)

Calibration

A well-calibrated model's confidence scores match actual frequencies — if the model says 80% confident, it should be right 80% of the time. Use Platt scaling or isotonic regression to calibrate raw logits.

Common Pitfalls

• Class imbalance: 95% majority class → always predict majority → 95% accuracy but 0% recall on minority. Use stratified splits, class weights, or SMOTE.
• Data leakage: text from same document in train and test; temporal leakage (future data in training).
• Label noise: inter-annotator disagreement — report Cohen's Kappa alongside accuracy.

Summary

• Naive Bayes with TF-IDF is a fast, interpretable baseline — great for high-dimensional sparse features
• Logistic Regression and LinearSVC with TF-IDF are often competitive with neural models at much lower cost
• BERT fine-tuning achieves SOTA for most classification tasks with >500 labeled examples per class
• Sentiment analysis spans document, sentence, and aspect level; VADER excels for social media without training
• Zero-shot classification via NLI (BART-MNLI) enables rapid deployment without any labeled data
• Evaluation: use F1/AUC-ROC over accuracy for imbalanced datasets; always check calibration

Next: Information Extraction & Named Entity Recognition — identifying structured facts from unstructured text.