Maximum Entropy Principle

Maximum Entropy model

One of the rule for statistical learning is maximum entropy. When we are learning a probabilistic model, the model with the maximum entropy is the best model. This principle gives us a heuristic for selecting the best model.

Simple patterns are compelling. When all the facts fit into a simple theory we are easily convinced that the pattern must be real, rather than random.

  • significance of simplicity : how unlikely is it that a simple pattern happen to be generated from a random process?

Human observers have a preference over simple and regular patterns. This idea has deep roots in psychology and philosophy. Concept learning, a term occurred usually in psychological literature, means the formation of generalization and the induction of categories.

  • Simplicity is the degree to which a given object can be effectively compressed without loss of information.
  • Simple theory are assigned higher prior probability than complex ones.

Kolmogorov complexity

  • the complexity value assigned to a pattern is the lowest value available in any description system.
  • a measure of the computability resources needed to specify the object.
  • the complexity of a string is the length of the shortest possible description of the string in some fixed universal description langugage.