Maximum Entropy model
One of the rule for statistical learning is maximum entropy. When we are learning a probabilistic model, the model with the maximum entropy is the best model. This principle gives us a heuristic for selecting the best model.
Simple patterns are compelling. When all the facts fit into a simple theory we are easily convinced that the pattern must be real, rather than random.
- significance of simplicity : how unlikely is it that a simple pattern happen to be generated from a random process?
Human observers have a preference over simple and regular patterns. This idea has deep roots in psychology and philosophy. Concept learning, a term occurred usually in psychological literature, means the formation of generalization and the induction of categories.
- Simplicity is the degree to which a given object can be effectively compressed without loss of information.
- Simple theory are assigned higher prior probability than complex ones.
Kolmogorov complexity
- the complexity value assigned to a pattern is the lowest value available in any description system.
- a measure of the computability resources needed to specify the object.
- the complexity of a string is the length of the shortest possible description of the string in some fixed universal description langugage.