Overfitting

Consider error of hypothesis $h$ over training data: $error_{train}(h)$ entire distribution $\cal{D}$ of data: $error_{\cal{D}}(h)$

Hypothesis $h \in H$ overfits training data if there is an alternative hypothesis $h' \in H$ such that

$$error_{train}(h) < error_{train}(h')$$

and

$$error_{\cal{D}}(h) > error_{\cal{D}}(h')$$