Search This Blog

Information Entropy Explained in 5 Minutes

Information Entropy Explained in 5 Minutes

Information entropy (also called Shannon Entropy) is a concept used in physics and information theory. Here's the scoop..

First define the system you're studying. This is not always easy, be careful you don't accidentally use a subset of the system.

Then define states of a system. A state is a set of parameters that tells you all you can know about the system for the purpose of your experiment. Assume your system can only be in one state at a time.

Make a large number of observations of the system, then use them to produce a good old fashioned probability, pi, the probability that if you make an observation the system is in state i. So for every state of the system you have a pi.

Now construct this crazy sum = p1*log(p1) + p2*log(p2) +... where the sum is over all the states of the system.

If the log is base 2 then (-1)*sum is called the "information entropy" of the system.

Where the heck did the log function come from?
log (p) is simply the power that 2 has to be raised by to make p. In other words it's x in this equation 2^x = p

Note that "information entropy" applies to a complete system, not individual states of a system.

Here's a simple example..

My system is a penny and a table.
I define the system to have 2 states.. penny lying stationary on the table with heads up or with tails up.

My experiment is to throw the penny and then observe which state results.

I throw the penny many times and make notes. It lands heads up 1% of the time and tails up 99% of the time (it's biased).

The crazy sum is 0.01*log(0.01) + 0.99*log(0.99) = 0.01*(-6.643856) + 0.99*(-0.0145) = -0.08079356

So the information entropy of the system is (-1)*(-0.08079356) = 0.08079356

Content written and posted by Ken Abbott