What Is Shannon Entropy?
Shannon entropy is H = −Σ pᵢ log pᵢ. It is zero when one outcome is certain and largest when all outcomes are equally likely. The unit depends on the logarithm base: bits for base 2, nats for base e and hartleys for base 10.
The calculator shows the maximum possible entropy for the number of outcomes, the normalised entropy H ÷ Hmax, the min-entropy, the Gini impurity and the perplexity. Reference points such as a fair coin (1 bit) and a certain outcome (0 bits) help you judge the value.
Below you will find the formulas, a variables table, a worked example and answers to common questions such as entropy in decision trees and how to convert between units.
How to Use the Shannon Entropy Calculator
- Choose whether you are entering probabilities or raw counts.
- Enter one value per outcome. Counts are converted to probabilities automatically.
- Pick the base: 2 for bits, e for nats.
- Compare your entropy with the maximum to see how close the distribution is to uniform.
Shannon Entropy Formula
Shannon entropy
Maximum entropy
Gini impurity
Perplexity
Shannon Entropy Variables Explained
| Symbol | Meaning | Notes |
|---|---|---|
| pᵢ | Probability of outcome i | Must sum to 1 |
| H | Shannon entropy | −Σ pᵢ log pᵢ |
| Hmax | Maximum entropy | log of the number of outcomes |
| Gini | Gini impurity | 1 − Σ pᵢ² |
| Perplexity | Effective number of outcomes | base^H |
Shannon Entropy Worked Example
Using the example values (press “Load example” above to try them):
- Normalise to probabilities p = value ÷ 14.
- Entropy H = −Σ p·log₂(p), skipping any p = 0.
- H = 0.940286 bits; the maximum for 2 outcomes is 1 bits.
Result: Shannon entropy H (bits) = 0.940286; Maximum possible entropy (log 2) = 1.
How to interpret entropy
Entropy is the average surprise you feel when you learn an outcome. Rare outcomes carry high surprise (−log p is large) but occur rarely; entropy weights surprise by probability. It reaches its maximum when all outcomes are equally likely and drops to zero when one outcome is certain.
| Distribution | Entropy (bits) |
|---|---|
| Fair coin (½, ½) | 1.000 |
| Biased coin (0.9, 0.1) | 0.469 |
| Fair die (6 × 1/6) | 2.585 |
| Four outcomes (½, ¼, ⅛, ⅛) | 1.750 |
| Certain outcome (1) | 0.000 |
Entropy in decision trees and machine learning
Decision tree algorithms such as ID3 and C4.5 choose splits that reduce entropy the most. Information gain is the entropy before a split minus the weighted entropy of the child nodes. CART uses Gini impurity, a closely related measure that is cheaper to compute. Cross-entropy, used as the loss in classification, is the entropy of the true distribution measured with the model’s probabilities.
Bits, nats and hartleys
The only difference is the logarithm base. Base 2 gives bits, natural logs give nats, and base 10 gives hartleys. Convert with 1 nat = 1.4427 bits and 1 hartley = 3.3219 bits.
Where Shannon Entropy Is Used
Information theory
Measuring the average information per outcome.
Decision-tree splits
Comparing entropy and Gini impurity of a node.
Comparing distributions
Seeing how evenly probability is spread across outcomes.
Coursework
Checking entropy calculations in bits, nats or hartleys.
Common Shannon Entropy Mistakes to Avoid
- Terms with p = 0 contribute 0 by convention.
- Entropy depends only on the probabilities, not on the labels or order of outcomes.
- With few samples the estimated entropy is biased low.
- Use normalised entropy to compare distributions with different numbers of outcomes.
Shannon Entropy Terms and Definitions
- Information gain
- Reduction in entropy achieved by splitting a data set on an attribute.
- Perplexity
- The effective number of equally likely outcomes, b^H.
- Min-entropy
- Entropy based only on the most likely outcome; a conservative measure used in security.
Shannon Entropy FAQs
What is Shannon entropy?
A measure of the average uncertainty in a probability distribution, calculated as −Σ p log p.
What is the entropy of a fair coin?
One bit. Two equally likely outcomes need one yes/no question to resolve.
Can entropy be negative?
For discrete distributions no; entropy is zero or positive. Differential entropy of continuous variables can be negative.
What is the difference between entropy and Gini impurity?
Both measure how mixed a distribution is. Entropy uses logarithms; Gini uses squared probabilities. They usually lead to similar decision trees.
How do I calculate entropy from counts?
Divide each count by the total to get probabilities, then compute −Σ p log₂ p.
Why does maximum entropy occur for uniform distributions?
When all outcomes are equally likely, you have no information favouring any of them, so uncertainty is at its highest.