Minimum Description Length Principle

Occam's razor: prefer the shortest hypothesis

MDL: prefer the hypothesis 3#3 that minimizes

64#64

where 65#65 is the description length of 66#66 under encoding 67#67

Example: 30#30 = decision trees, 5#5 = training data labels

• 68#68 is # bits to describe tree 3#3
• 69#69 is # bits to describe 5#5 given 3#3
  • Note 70#70 if examples classified perfectly by 3#3. Need only describe exceptions
• Hence 71#71 trades off tree size for training errors