Knowledge as entropy reduction: what it means to know
Click each concept to reveal definitions, examples, and connections
INFORMATION IS SURPRISE REDUCTION
H = -Σ p(x) log p(x)
Shannon entropy: average surprise in a message. Learning reduces H.
KNOWLEDGE ▶
as Constraint
To know is to reduce possible states.
It could have been otherwise, but isn't.
Knowledge is the elimination of possibilities. Before measurement, a particle could be anywhere. After measurement, it is here. Each bit of knowledge halves the state space. A theory that constrains predictions more tightly, ruling out more of what could be but isn't, is a better theory. This is why science progresses by making falsifiable claims: each surviving test is another constraint, another reduction of entropy.
EXAMPLE
A medical diagnosis: from "could be anything" (high H) to "it is this specific condition" (low H). The diagnosis IS the entropy reduction.
IGNORANCE ▶
as Entropy
Not knowing = high H. Many states compatible with observations.
Maximum entropy = prior
Ignorance is literally high entropy: many possible states are equally consistent with what you observe. The maximum entropy principle says that when you know nothing, assign equal probability to all possibilities. This isn't a failure; it's the honest starting position. Jaynes showed that the maximum entropy distribution is the uniquely correct prior: it encodes exactly what you know and nothing more.
EXAMPLE
Before rolling a die, each face has probability 1/6. That uniform distribution IS your ignorance, expressed mathematically. H is at maximum.
LEARNING ▶
as Compression
Find patterns that compress the data.
Minimum description length = best theory
Learning is the discovery of compressible structure. A dataset that seems random (high H) turns out to follow a pattern; now the same data can be described more concisely. Kolmogorov complexity: the length of the shortest program that produces the data. Minimum Description Length: the best model balances fit against complexity. Science, machine learning, and biological cognition all do the same thing: they compress experience into reusable patterns that reduce future surprise.
EXAMPLE
Newton's F=ma compresses millions of observations into three symbols. The compression IS the understanding. That is what a theory does: it makes the universe smaller.
IMPLICATIONS FOR KNOWING
Objectivity:Maximally constrained by evidence (minimum entropy given data)
Parsimony:Prefer theories that compress more (Occam's razor as entropy)