Information theory
Information theory
We define a quantity I (the information) contained in a prob- ability distribution by requiring that this quantity has the following properties (Khinchine) - The information depends only on the probability distribution - The uniform distribution contains the minimum information. - If we enhance the sample space with impossible events the information does not increase. - Information is additive It is possible to show, starting from these axioms, that the information contained in a probability distribution that has N possible outcomes in the sample space, Ω, that have probabilities given by the vector p, is equal to: I(p)=kN∑i=1pilnpi
Syllabus Aims
- You should be able to state the Khinchine axioms of information theory
- You should be able to give an expression for the information contained in a uniform distribution.
- You should be able to give an expression for the information contained in a non-uniform distribution.
- You should be able to explain how entropy and information are related.
- You should be able to write an expression for the entropy of a uniform distribution
Description and link | Module | Author | ||
| A video on the Khinchine axioms for information theory in which the information contained in a uniform distribution is derived. | PHY9038/AMA4004 | G. Tribello | ||
| A video in which the information contained in a non-uniform distribution is derived. | PHY9038/AMA4004 | G. Tribello |
Contact Details
School of Mathematics and Physics,
Queen's University Belfast,
Belfast,
BT7 1NN
Email: g.tribello@qub.ac.uk
Website: mywebsite