Phonology & Speech Processing

Phonology in NLP

Phonology is the study of the sound systems of languages — how phonemes (the smallest units of sound) combine and interact to convey meaning.

Phonemes

A phoneme is the smallest unit of sound that can distinguish meaning in a language. English has ~44 phonemes represented by the International Phonetic Alphabet (IPA):

Phoneme	Example Word	IPA
/p/	pin	/pɪn/
/b/	bin	/bɪn/
/θ/	think	/θɪŋk/
/ð/	this	/ðɪs/
/ŋ/	sing	/sɪŋ/

Allophones are variant pronunciations of the same phoneme that don't change meaning — e.g., the "p" in "pin" (aspirated) vs "spin" (unaspirated) are allophones of /p/.

Why Phonology Matters for NLP

• Speech-to-text (ASR): maps audio signals → phonemes → words
• Text-to-speech (TTS): maps text → phonemes → audio
• Pronunciation dictionaries: CMU Pronouncing Dictionary maps words to phoneme sequences
• Spell checking: phoneme-based models catch "there/their/they're" errors
• Rhyme detection: poetry and lyrics analysis

Prosody

Prosody captures the rhythm, stress, and intonation of speech:

• Pitch (F0): rising = question, falling = statement
• Duration: stressed syllables are longer
• Energy: louder for emphasis

Modern TTS systems must model prosody to sound natural — flat prosody sounds robotic.

Automatic Speech Recognition (ASR)

Feature Extraction: MFCC

Raw audio (waveform) is first converted to Mel-frequency cepstral coefficients (MFCCs) — a compact representation of the spectral shape that mimics human auditory perception:

1. Frame the signal into 20–25ms overlapping windows
2. Apply FFT to get the frequency spectrum
3. Apply a Mel filter bank (non-linear frequency scale matching human perception)
4. Take the log of the energies
5. Apply DCT to decorrelate → MFCC features (typically 13–40 coefficients)

$\text{Mel}(f) = 2595 \cdot \log_{10}\left(1 + \frac{f}{700}\right)$

The Classic ASR Pipeline

Audio Waveform
  ↓  Feature Extraction (MFCC)
Acoustic Features
  ↓  Acoustic Model (HMM-GMM or DNN)
Phoneme Probabilities
  ↓  Decoder (Viterbi + Language Model)
Word Sequence (Transcript)

Acoustic Model: maps audio features to phoneme/character probabilities Language Model: provides prior probability of word sequences $P(w_1 w_2 \ldots w_n)$ Decoder: finds the most likely word sequence:

$W^* = \arg\max_W P(A|W) \cdot P(W)$

where $P(A|W)$ = acoustic likelihood, $P(W)$ = language model probability.

Modern ASR Approaches

Approach	Architecture	Key Idea
CTC (Connectionist Temporal Classification)	Encoder + linear	Allows variable-length alignment without explicit segmentation
LAS (Listen Attend Spell)	Encoder-Decoder + Attention	Attention-based seq2seq; no need for separate LM
Whisper (OpenAI)	Transformer encoder-decoder	Trained on 680k hours of multilingual audio; SOTA zero-shot
wav2vec 2.0 (Meta)	CNN + Transformer + CTC	Self-supervised pre-training on raw audio

CTC Loss

CTC allows training without frame-level alignment by marginalizing over all valid alignments:

$\mathcal{L}_{CTC} = -\log P(y | x) = -\log \sum_{\pi \in \mathcal{B}^{-1}(y)} \prod_t P(\pi_t | x)$

The blank token ε handles repeated characters and silence: "HELLO" → "HH-E-LL-L-OO" → collapse → "HELLO"

# ─── Whisper ASR (OpenAI) ────────────────────────────────────
# pip install openai-whisper
import whisper
import numpy as np

# Load model (tiny/base/small/medium/large — trade speed vs accuracy)
model = whisper.load_model("base")

# Transcribe an audio file
result = model.transcribe("audio.wav", language="en", task="transcribe")
print("Transcript:", result["text"])
print("Language detected:", result["language"])

# With word-level timestamps
result_ts = model.transcribe("audio.wav", word_timestamps=True)
for segment in result_ts["segments"]:
    print(f"[{segment['start']:.2f}s - {segment['end']:.2f}s]: {segment['text']}")

# Translation to English (any language → English)
result_translated = model.transcribe("audio_spanish.wav", task="translate")
print("Translation:", result_translated["text"])

# ─── MFCC Feature Extraction ─────────────────────────────────
# pip install librosa
import librosa
import librosa.display

# Load audio
y, sr = librosa.load("audio.wav", sr=16000)  # 16kHz mono
print(f"Audio shape: {y.shape}, Sample rate: {sr}")

# Extract MFCCs
mfccs = librosa.feature.mfcc(y=y, sr=sr, n_mfcc=13, hop_length=512, n_fft=2048)
print(f"MFCC shape: {mfccs.shape}")  # (13, time_frames)

# Delta features (velocity and acceleration of MFCCs)
mfcc_delta = librosa.feature.delta(mfccs)
mfcc_delta2 = librosa.feature.delta(mfccs, order=2)
mfcc_full = np.vstack([mfccs, mfcc_delta, mfcc_delta2])  # 39 features
print(f"MFCC + deltas shape: {mfcc_full.shape}")

# Mel spectrogram
mel_spec = librosa.feature.melspectrogram(y=y, sr=sr, n_mels=80)
mel_db = librosa.power_to_db(mel_spec, ref=np.max)
print(f"Mel spectrogram shape: {mel_db.shape}")

# ─── HuggingFace ASR Pipeline ────────────────────────────────
from transformers import pipeline

# Wav2Vec2 for ASR
asr = pipeline("automatic-speech-recognition",
               model="facebook/wav2vec2-base-960h",
               device=0)  # GPU; -1 for CPU

result_hf = asr("audio.wav")
print("\nHuggingFace ASR:", result_hf["text"])

# Whisper via HuggingFace (more control)
asr_whisper = pipeline("automatic-speech-recognition",
                       model="openai/whisper-base",
                       chunk_length_s=30,
                       stride_length_s=5)
result_w = asr_whisper("long_audio.wav", return_timestamps=True)
print("Whisper chunks:", result_w["chunks"][:3])

Text-to-Speech (TTS)

TTS converts text into natural-sounding speech. Modern TTS is a two-stage pipeline:

Text
  ↓  Text Analysis (normalization, G2P, prosody prediction)
Linguistic Features (phonemes, stress, duration)
  ↓  Acoustic Model (predicts mel spectrogram)
Mel Spectrogram
  ↓  Vocoder (converts spectrogram → waveform)
Audio Waveform

Grapheme-to-Phoneme (G2P)

G2P converts written text to phoneme sequences: "read" → /rɛd/ (past) or /riːd/ (present) — context-dependent!

TTS Systems Evolution

Generation	System	Approach
Concatenative	Festival, MBrola	Splices recorded audio segments
Statistical	HMM-TTS	Hidden Markov Model parameters
Neural	Tacotron 2	Seq2seq with attention → mel → WaveNet
Neural (fast)	FastSpeech 2	Non-autoregressive, 10x faster
Neural (E2E)	VITS, NaturalSpeech	End-to-end; no separate vocoder
Zero-shot	VALL-E (Microsoft)	3-second voice cloning prompt

Vocoders

Vocoders convert mel spectrograms back to raw audio waveforms:

• WaveNet: autoregressive, very slow, high quality
• WaveGlow: flow-based, fast, parallel generation
• HiFi-GAN: GAN-based, real-time, state-of-the-art quality

ASR Evaluation Metrics

Word Error Rate (WER) — the primary ASR metric:

$WER = \frac{S + D + I}{N}$

where $S$ = substitutions, $D$ = deletions, $I$ = insertions, $N$ = total words in reference.

• WER = 0%: perfect transcription
• Human WER: ~5–8% for conversational speech
• Whisper large-v3: ~3–5% WER on standard benchmarks

Summary

• Phonemes are the minimal units of sound; ~44 in English represented in IPA
• ASR pipeline: MFCC features → acoustic model → decoder with language model
• CTC enables ASR training without frame-level alignment via blank token collapsing
• Whisper: a single transformer trained on 680k multilingual hours — SOTA zero-shot ASR
• TTS pipeline: text → G2P → acoustic model (mel spectrogram) → vocoder (waveform)
• WER is the primary ASR metric: lower is better; human-level ≈ 5–8%

Next: Morphology & Lexical Analysis — understanding word structure and meaning.