The shape of an argument: a field guide to statistics

Every statistical method answers a narrow question. The trouble is rarely the math — it is picking the right question for the data in front of you.

This guide walks through the toolkit a working analyst reaches for again and again: comparing groups, checking assumptions, reading relationships, watching data move through time, simplifying a crowded feature space, and making sure a comparison between two groups is actually a fair one. Each entry covers what the method is for, how it reasons, a concrete example, and where it fits next to its neighbors.

1. Testing for real differences between groups
2. Checking your assumptions first
3. Reading relationships between variables
4. Getting to know a dataset
5. Watching data move through time
6. Simplifying and explaining a model
7. Making sure a comparison is fair

Testing for real differences between groups

The most common question in applied statistics: are these two (or more) groups actually different, or does it just look that way in this sample?

Statistical significance testing

Purpose: decide whether an observed difference reflects a real effect or could plausibly be chance.

Every test below is a variation on the same logic. You assume, for the sake of argument, that there is no real effect (the null hypothesis), then ask how surprising your data would be if that assumption were true. If the data would be very surprising — conventionally, less than a 5% chance of occurring by luck — you reject the assumption and call the result statistically significant.

Sits next to: baseline validation, ANOVA, independent t-test.

Independent t-test

Purpose: compare the means of exactly two independent groups.

The t-test compares the gap between two group averages against how much those averages would naturally wobble from sample to sample. It assumes both groups are roughly normally distributed and were sampled independently — no shared subjects, no repeated measurements on the same people.

Sits next to: Mann-Whitney U (when normality fails), Hotelling’s $T^2$ (when comparing several variables at once).

Mann-Whitney U test

Purpose: compare two independent groups without assuming a normal distribution.

Rather than comparing means directly, this test ranks all observations from both groups together and checks whether one group’s ranks are systematically higher. That makes it robust to skew, outliers, and ordinal data — useful whenever the t-test’s normality assumption is shaky.

Sits next to: independent t-test, Kruskal-Wallis (its 3+ group counterpart).

ANOVA (analysis of variance)

Purpose: test whether at least one of three or more group means differs from the rest.

ANOVA compares the variation between group means to the variation within each group. If groups differ more from each other than individual points differ within a group, that is evidence of a real effect somewhere among them. A significant ANOVA says something differs — it does not say which pair.

Sits next to: Kruskal-Wallis, Tukey HSD (the natural follow-up).

Kruskal-Wallis test

Purpose: ANOVA’s rank-based counterpart for non-normal or ordinal data.

Like Mann-Whitney extends the t-test, Kruskal-Wallis extends ANOVA to situations where the normality assumption does not hold, by comparing ranks across all groups instead of raw values.

Sits next to: ANOVA, Mann-Whitney U.

Tukey HSD (honestly significant difference)

Purpose: after ANOVA finds a difference somewhere, identify exactly which group pairs differ.

Tukey HSD runs every pairwise comparison among the groups while controlling the overall false-positive rate — something that running plain t-tests pair by pair would not do. It is the standard second step once an ANOVA comes back significant.

Sits next to: ANOVA.

Chi-square test

Purpose: test whether two categorical variables are related, or whether observed category counts match an expected pattern.

Chi-square compares observed counts in a table of categories against the counts you’d expect if the variables were unrelated. A large gap between observed and expected counts is evidence of a real association.

Sits next to: propensity score matching.

Baseline validation

Purpose: confirm that feature distributions in a new sample still match a trusted reference (baseline) before trusting further analysis.

This is not a single test but a practice: applying t-tests, ANOVA, chi-square, and similar tools systematically across many features to catch drift, sampling bugs, or class imbalance before they quietly bias downstream results.

Sits next to: statistical significance testing, ANOVA, independent t-test.

Checking your assumptions first

Every parametric test in the previous section assumes something about how the data is shaped. These two methods check that assumption before you rely on it.

Shapiro-Wilk test

Purpose: test whether a sample plausibly came from a normally distributed population.

It measures how closely the sorted sample values track what you’d expect under a perfect normal distribution. It is generally regarded as the most powerful normality test for small-to-medium samples, which is why it is the default first check.

Sits next to: D’Agostino’s test.

D’Agostino’s test

Purpose: an alternative normality check, based specifically on a distribution’s skewness and kurtosis (its tail weight).

Where Shapiro-Wilk gives one overall verdict, D’Agostino’s test isolates why a distribution deviates from normal — lopsided (skewed) or heavy-tailed (kurtotic) — which can point more directly at the right fix, such as a log transform for skew.

Sits next to: Shapiro-Wilk.

Reading relationships between variables

Correlation methods describe how two variables move together — a different question from whether groups differ, and a common source of confusion with causation.

Correlation analysis

Purpose: quantify how strongly, and in what direction, pairs of numeric variables move together.

Usually shown as a correlation matrix — every variable against every other — this is exploratory: a first pass to spot which relationships are worth investigating further, not a hypothesis test on its own.

Sits next to: Pearson correlation, Spearman rank correlation.

Pearson correlation

Purpose: measure the strength of a linear relationship between two continuous variables, on a scale from $-1$ to $1$.

It assumes both variables are roughly continuous and normally distributed, and it specifically captures straight-line relationships — a strong curved relationship can score near zero even though the variables are clearly connected.

Sits next to: Spearman rank correlation, correlation analysis.

Spearman rank correlation

Purpose: measure whether two variables move consistently in the same (or opposite) direction, even if not in a straight line.

Spearman correlates the ranks of the values rather than the raw numbers, so it captures any consistently increasing or decreasing relationship — linear or not — and is far less sensitive to outliers than Pearson.

Sits next to: Pearson correlation, correlation analysis.

Getting to know a dataset

Before testing anything, look at what you have. These methods describe a dataset rather than test a hypothesis about it.

Descriptive statistics

Purpose: summarize a dataset’s central tendency and spread — mean, median, mode, standard deviation, min/max — in a handful of numbers.

This is usually the first thing computed on any new dataset. It is not a test of anything; it is the baseline vocabulary every later analysis builds on.

Sits next to: distribution analysis.

Distribution analysis

Purpose: examine the shape of a variable’s distribution — skew, tails, multiple peaks, outliers — beyond single summary numbers.

Where descriptive statistics reduce a variable to a few numbers, distribution analysis looks at the full shape, typically through histograms, density plots, and box plots, to catch patterns that summary numbers alone would hide.

Sits next to: descriptive statistics.

Missing data analysis

Purpose: identify how much data is missing, whether it is missing in a random or patterned way, and choose a sound way to handle it.

The key question is not just “how much is missing” but “why.” Missingness that correlates with other variables (say, high earners skipping an income field) can bias results if handled carelessly — a plain average-fill can quietly distort the analysis.

Sits next to: baseline validation.

Watching data move through time

Time series data breaks the independence assumption most tests rely on — today’s value depends on yesterday’s. These methods are built for that.

Time series decomposition

Purpose: split a time series into its trend, seasonal pattern, and leftover (residual) noise.

Separating these components makes each easier to reason about on its own — a rising trend obscured by strong weekly seasonality is much clearer once the two are pulled apart.

Sits next to: trend analysis, ARIMA.

Trend analysis

Purpose: detect and quantify the long-term direction in a time series, separate from short-term noise or seasonality.

This can range from a simple moving average to formal trend-significance tests, but the goal is the same: is this genuinely going up (or down) over time, or is that impression coming from a few noisy recent points?

Sits next to: time series decomposition.

ARIMA (autoregressive integrated moving average)

Purpose: forecast a single time series by modeling how past values and past errors predict future ones.

ARIMA combines three ideas: using recent past values to predict the next one (autoregression), differencing the series to remove trend (integration), and using past forecast errors to correct future ones (moving average). It is a strong, well-understood default for a single series without external drivers.

Sits next to: SARIMAX, exponential smoothing.

SARIMAX (seasonal ARIMA with exogenous variables)

Purpose: extend ARIMA to handle recurring seasonal cycles and incorporate outside predictor variables.

Plain ARIMA struggles with strong seasonality and cannot use external information. SARIMAX adds a seasonal component and lets you feed in exogenous variables — like holidays or marketing spend — that plausibly drive the series alongside its own history.

Sits next to: ARIMA, exponential smoothing.

Exponential smoothing

Purpose: forecast a time series by weighting recent observations more heavily than older ones.

Simpler and more interpretable than ARIMA, it is a strong choice when a series is fairly stable and you want a quick, robust forecast without fitting a more complex model.

Sits next to: ARIMA, SARIMAX.

Simplifying and explaining a model

When there are too many variables to reason about directly, or a model’s predictions need a plain-language explanation, these methods bridge the gap.

PCA (principal component analysis)

Purpose: compress many correlated variables into a smaller set of components that preserve most of the original information.

PCA finds the directions in the data along which it varies the most, and re-expresses the dataset along those directions. Often the first two or three components capture most of what matters, letting you visualize or model dozens of variables through just a few.

Sits next to: feature importance analysis.

Feature importance analysis

Purpose: rank which input variables contribute most to a model’s overall predictions.

Most modern models (tree ensembles especially) can report how much each feature reduced prediction error across the whole training set — a global view of what the model is paying attention to.

Sits next to: SHAP, PCA.

SHAP (SHapley Additive exPlanations)

Purpose: explain a single prediction by attributing it to each input feature’s specific contribution.

Where feature importance gives a model-wide average, SHAP (built on a game-theory concept called Shapley values) answers “why did the model predict this for this customer” — a per-prediction explanation rather than a per-model one.

Sits next to: feature importance analysis.

Making sure a comparison is fair

When groups cannot be randomly assigned — most real-world studies — these methods check and correct for the fact that the groups may differ for reasons that have nothing to do with what you are testing.

Propensity score matching

Purpose: make treatment and control groups comparable in an observational study by matching subjects with similar odds of receiving the treatment.

Without random assignment, people who chose a treatment often differ systematically from those who did not — a selection bias that can masquerade as a treatment effect. Propensity score matching models the probability of receiving treatment from observed covariates, then pairs similar-probability subjects across groups so the comparison isolates the treatment itself.

Sits next to: SMD, propensity AUC, chi-square.

SMD (standardized mean difference)

Purpose: quantify how balanced two groups are on a given covariate, in units that are comparable across variables of different scales.

SMD expresses the gap between two group means as a fraction of their pooled standard deviation. As a rule of thumb, values under 0.1 indicate good balance, and values above 0.25 flag a covariate that likely still needs attention.

Sits next to: propensity AUC, propensity score matching.

Propensity AUC (area under curve)

Purpose: check overall covariate balance by seeing how well a model can still predict group membership after matching.

If treatment and control are well-balanced, a model trying to predict which group someone belongs to (from the same covariates used for matching) should perform barely better than a coin flip. An AUC near 0.5–0.55 signals good balance; anything much higher means the groups are still distinguishable, and the matching needs revisiting.

Sits next to: SMD, propensity score matching.

MMD (maximum mean discrepancy)

Purpose: test whether two groups follow the same overall distribution, not just whether their means differ.

Using kernel methods, MMD compares entire distributions — capturing differences in spread, shape, and tails that a mean-based test like SMD would miss entirely if the means happened to line up.

Sits next to: SMD, Hotelling’s $T^2$.

Hotelling’s $T^2$

Purpose: the multivariate generalization of the t-test — compare two groups across several variables simultaneously, not one at a time.

Testing five variables one by one with separate t-tests inflates the chance of a false positive somewhere and ignores how the variables relate to each other. Hotelling’s $T^2$ tests all of them jointly in a single, statistically coherent comparison.

Sits next to: ANOVA, independent t-test, MMD.

References and further reading

1. Montgomery, D. C. Design and Analysis of Experiments. Wiley. — Core reference for ANOVA, post-hoc testing (Tukey HSD), and experimental design.
2. Wasserman, L. All of Statistics: A Concise Course in Statistical Inference. Springer. — Broad, rigorous coverage of hypothesis testing, estimation, and non-parametric methods.
3. Shapiro, S. S., & Wilk, M. B. (1965). “An analysis of variance test for normality (complete samples).” Biometrika, 52(3–4), 591–611.
4. D’Agostino, R. B., Belanger, A., & D’Agostino Jr, R. B. (1990). “A suggestion for using powerful and informative tests of normality.” The American Statistician, 44(4), 316–321.
5. Hyndman, R. J., & Athanasopoulos, G. Forecasting: Principles and Practice. OTexts (open access) — otexts.com/fpp3. — Time series decomposition, ARIMA, SARIMAX, and exponential smoothing.
6. Jolliffe, I. T. Principal Component Analysis. Springer Series in Statistics.
7. Lundberg, S. M., & Lee, S.-I. (2017). “A Unified Approach to Interpreting Model Predictions.” Advances in Neural Information Processing Systems (NeurIPS). — Introduces SHAP.
8. Rosenbaum, P. R., & Rubin, D. B. (1983). “The central role of the propensity score in observational studies for causal effects.” Biometrika, 70(1), 41–55.
9. Austin, P. C. (2011). “An introduction to propensity score methods for reducing the effects of confounding in observational studies.” Multivariate Behavioral Research, 46(3), 399–424. — SMD thresholds and propensity score diagnostics.
10. Gretton, A., Borgwardt, K. M., Rasch, M. J., Schölkopf, B., & Smola, A. (2012). “A Kernel Two-Sample Test.” Journal of Machine Learning Research, 13(1), 723–773. — Maximum Mean Discrepancy (MMD).
11. Hotelling, H. (1931). “The generalization of Student’s ratio.” Annals of Mathematical Statistics, 2(3), 360–378.