Interactive guide
Every split answers one question: how much cleaner did that make things?
A decision tree grows by repeatedly asking "which single feature and threshold, if I split on it, leaves the two resulting groups as pure as possible?" Entropy and the Gini index are the two usual ways to measure "pure," and information gain is just how much purer a split makes things. All three below, live.
Impurity, then gain, then a tree
Entropy: H = −Σ pi log₂(pi). Gini: G = 1 − Σ pi². Both are 0 when a node is pure (every sample the same class) and both peak when classes are perfectly balanced; they just weight the in-between mixtures slightly differently.
Information gain for a candidate split is the parent's impurity minus the sample-weighted impurity of the two children it produces. The split with the highest gain is the one the tree actually picks at that node.
Grow that logic recursively: split, then split each child, then each of theirs, stopping at pure nodes or a depth limit, and you have a decision tree. Nothing about the tree is decided in advance; every split is whichever one the impurity math favors.
How one split gets picked
-
1
Measure the parent's impurity
Entropy or Gini of the current node's class mix, before any split.
-
2
Try every feature and threshold
For each candidate split, compute the weighted impurity of the two resulting groups.
-
3
Keep the one with the highest gain
Parent impurity minus weighted child impurity: biggest drop wins.
-
4
Recurse on each child
Repeat inside each new node until it's pure or a stopping rule kicks in.
Part 1: How "impure" is a node? Entropy vs. Gini
- Entropy
- 1.000
- Gini
- 0.500
This node's 20 samples
Entropy & Gini vs. class proportion
Entropy Gini
Part 2: Finding the best split: information gain vs. threshold
20 students, hours studied vs. pass/fail. Drag the threshold and watch both child nodes, and the resulting gain, update live.
Students split by this threshold
fail pass
Information gain vs. threshold
gain (entropy) gain (Gini)
—
- Left node
- —
- Right node
- —
- Information gain
- —
Part 3: Growing a full tree, two ways
36 points, 3 classes, 2 features. Same greedy algorithm, same data; only the impurity measure changes. Click any node to see exactly what it split on and inspect its region.
The tree
The data, partitioned
Class A Class B Class C
Click any node in the tree to inspect it.
On this dataset, entropy and Gini don't even agree on the root split, different threshold, different shape of tree entirely. That's the honest picture: the two criteria almost always rank splits similarly, but "almost always" isn't "always," and neither one is simply the correct answer. In practice the choice rarely changes a tree's accuracy much; Gini is slightly cheaper to compute (no logarithms), which is why it's the default in many libraries.
The jargons
Entropy and Gini are two competing ways to score how mixed-up a node’s classes are; information gain is what a tree actually optimizes with whichever one it’s using, one greedy split at a time.
H = −Σ p log₂(p), measured in bits, 0 for a pure node, maximum (1.0 for two classes) at a perfect 50/50 split.
G = 1 − Σ p²: the probability a randomly picked pair of samples from the node would have different classes if labeled randomly. 0 for a pure node, 0.5 at 50/50 for two classes.
Parent impurity minus the sample-weighted impurity of the children a split produces. The split a tree actually picks is whichever maximizes this.
A node the tree stops splitting, either because it's pure, or a stopping rule (max depth, minimum samples) was hit. Its prediction is its majority class.
At each node, the tree picks whichever single split looks best right now; it never looks ahead to whether a different split might set up a better one two levels down.
Classification and Regression Trees: the standard algorithm this page implements: binary splits, greedy, grown until a stopping rule.