## Maximum Entropy model

One of the rule for statistical learning is *maximum entropy*. When we are learning a probabilistic model, the model with the *maximum entropy* is the best model. This principle gives us a heuristic for selecting the best model.

Simple patterns are compelling. When all the facts fit into a simple theory we are easily convinced that the pattern must be real, rather than random.

- significance of simplicity : how unlikely is it that a simple pattern happen to be generated from a random process?

Human observers have a preference over simple and regular patterns. This idea has deep roots in psychology and philosophy. Concept learning, a term occurred usually in psychological literature, means the formation of generalization and the induction of categories.

- Simplicity is the degree to which a given object can be effectively compressed without loss of information.
- Simple theory are assigned higher prior probability than complex ones.

Kolmogorov complexity

- the complexity value assigned to a pattern is the lowest value available in any description system.
- a measure of the computability resources needed to specify the object.
- the complexity of a string is the length of the shortest possible description of the string in some fixed universal description langugage.