undefined | Better HN

0 pointsrsync8y ago0 comments

"A bit isn't a divisible unit"

What ? Sure it is. When you measure the information content (or, the entropy) of a message, you very frequently get non-integer numbers of bits per (character/unit/message/whatever).

Written english, for instance, has 1.46 bits of information per character.

Or 146 cb.

0 comments

kev0098y ago

That is expressing a ratio. A more concrete example, .46 a person doesn't exist even if that ratio is useful for expressing statistics.

drdeca8y ago

I think "gaining half a bit of information" about something can correspond to stuff that lets you update your probability distribution about it?

I'm not sure.

kristjankalm8y ago

yes, but your gain is still an integer number. the non-integer ratio is just a statistic (gain per number of units).

2 more replies

akvadrako8y ago

Or to express it another way, a variable with 3 possible states has 0.5 bits more capacity than one with 2.

stephencanon8y ago

0.585... bits more capacity, but yeah (bits = log2(states)).

PeCaN8y ago

> you very frequently get non-integer numbers of bits per (character/unit/message/whatever)

That'd be an average, and it's like when you have 2.58 people per household. Presumably most people do not keep about 6/10 of a person around.

The point of a bit (in information theory) is that it is the smallest possible (read: not divisible) unit of information.

j / k navigate · click thread line to collapse