Activation Functions Angle Sum Formulas Backpropagation Bias-Variance Tradeoff Central Limit Theorem Central Tendency Chain Rule Class Imbalance Conditional Probability Confidence Intervals Cross-Validation DBSCAN Decision Tree Derivatives & Integration Determinant as Area Scaling Dispersion Distance Metrics Dot Product & Projection Ensemble Learning Eigenvalues & Eigenvectors Empirical Rule (68-95-99.7) Expected Value & Variance Feature Scaling Gradient Descent Hyperparameter Tuning Hypothesis Testing K-Means Clustering K-Nearest Neighbors L1 vs. L2: The Geometry of Sparsity Law of Large Numbers Law of Sines & Cosines Limits Linear Discriminant Analysis Linear Regression Logistic Regression Markov Chains Matrix Transformations Multi-Layer Perceptron Naive Bayes Optimizers Race Partial Derivatives & Gradient Perceptron Precision-Recall Curve & F1 Principal Component Analysis Regularization ROC & AUC Sampling Techniques Singular Value Decomposition SVM & the Margin Systems of Linear Equations Taylor Series Unit Circle Vanishing/Exploding Gradients

Interactive guide

The knob you set before the model learns anything

A parameter (a regression coefficient, a tree's split thresholds, a cluster's centroid) is learned from data. A hyperparameter (k, max depth, alpha, n_components) is chosen before that learning starts, and the data never gets a vote. The only way to choose one well is to try several values and measure what happens. Below, the same sweep happens six times, across a regressor, a classifier, a tree, a regularized regression, a clusterer, and a dimensionality reducer.

Two ways to score a sweep

When there are labels to check against, split the data first: fit on training points, score on validation points the fit never saw. Training error almost always keeps improving as a model gets more flexible: more neighbors' worth of nuance, deeper splits, less penalty. Validation error doesn't: it improves for a while, then turns around once the model starts fitting noise instead of signal. The hyperparameter at that turning point, not the one with the lowest training error, is the one to keep.

Clustering and dimensionality reduction have no labels to validate against: there's no "correct" cluster or "correct" projection to check a guess against. The stand-in is a diminishing-returns curve instead: how much does one more cluster shrink inertia, how much variance does one more component actually explain. The elbow where returns flatten out plays the same role the validation minimum plays elsewhere.

Reading every panel below

1
Drag the slider

Left chart redraws the actual model (its fit, boundary, clusters, or projection) at that hyperparameter value.
2
Watch the middle chart

Training score (teal) vs. validation score (orange), or an elbow curve for the two unsupervised panels, with a marker tracking the slider.
3
Find the green dot

Marks the best validation score anywhere on the sweep: the value an actual grid search would return.

1 · Regression: K-Nearest Neighbors hyperparameter: k

26 points, split into training (filled) and validation (hollow); a prediction is just the average y of the k nearest training x's. k=1 has to pass exactly through every training point it's asked about, since its own nearest neighbor at distance 0 is itself.

scikit-learnknn = KNeighborsRegressor(n_neighbors=5)

train validation line = k-NN prediction across x.

train MSE val MSE

Small k memorizes (near-zero train error, high val error). Large k averages over too much of the line and both errors climb.

Neighbors (k) —

—

Train MSE: —
Val MSE: —

—

2 · Classification: K-Nearest Neighbors hyperparameter: k

90 points, two overlapping classes, split into training and validation. A query point's class is decided by a vote among its k nearest training neighbors; the shading below is that vote, computed at every point in the plane.

scikit-learnknn = KNeighborsClassifier(n_neighbors=5)

class 1 class 0 hollow = validation shading = k-NN vote.

train acc val acc

k=1 fits training data perfectly, but perfectly memorizing which exact points were seen isn't the same as generalizing to new ones.

Neighbors (k) —

—

Train acc: —
Val acc: —

—

3 · Classification: Decision Tree hyperparameter: max_depth

90 points along one noisy feature, four alternating true regions. Each extra level of depth lets the tree cut the line into twice as many pieces, eventually into pieces built around single mislabeled points instead of the real regions.

scikit-learndtc = DecisionTreeClassifier(criterion="gini", random_state=100, max_depth=3, min_samples_leaf=5)

class 1 class 0 hollow = validation dashed = split thresholds.

train acc val acc

Training accuracy can only rise or hold as depth grows: every extra split can only carve training data more finely, never less.

Max depth —

—

Train acc: —
Val acc: —

—

4 · Regression: Ridge Regression hyperparameter: alpha (α)

16 noisy points fit with a degree-15 polynomial, nearly as many coefficients as points, wildly overfit at α=0. Turning up α leans on every coefficient proportionally, trading training accuracy for a curve that actually generalizes, until it leans so hard the curve goes flat again.

scikit-learnridge = Ridge(alpha=1.0)

train test dashed = true curve solid = current fit.

train MSE test MSE

Log-spaced alpha, left to right.

α index —

—

Train MSE: —
Test MSE: —

—

5 · Clustering: K-Means hyperparameter: k

60 unlabeled points. No validation set is possible here: there's no "correct" cluster to check a guess against. Instead, watch inertia (total squared distance to each point's own centroid): it never stops shrinking as k grows, so the question isn't "is it still improving" but "is it still improving by much."

scikit-learnkmeans = KMeans(n_clusters=3, random_state=42, n_init=10)

Colors = current cluster assignment. Diamonds = centroids, from a deterministic farthest-point start.

inertia (best of several restarts)

Look for the elbow: the k where the curve stops dropping steeply and starts flattening out.

Clusters (k) —

—

Inertia (this run): —
Drop from k−1: —

—

6 · Dimensionality Reduction: PCA hyperparameter: n_components

50 students, 4 exam scores each (Math, Physics, Chemistry, Writing), built from just two underlying abilities plus noise, so most of the real variation lives in far fewer than 4 dimensions. n_components picks how many of the 4 principal components to keep; the rest is treated as noise and thrown away.

scikit-learnpca = PCA(n_components=2)

PC1 (STEM ability) vs. PC2 (verbal ability).

kept dropped

Bar height = variance that component alone explains. Label on top = cumulative, kept components only.

n_components —

—

Variance kept: —
Reconstruction MSE: —

—

Same shape, six times: the panel doing best on data it fit is almost never the panel doing best on data it didn't. That gap , not raw training performance, is what every hyperparameter search is actually looking for.