Gradient Descent

To understand, consider simpler linear unit, where

$$o = w_{0} + w_{1}x_1 + \cdots + w_n x_n$$

Let's learn $w_{i}$'s that minimize the squared error

$$E[\vec{w}] \equiv \frac{1}{2}\sum_{d \in D}(t_{d} - o_{d})^{2}$$

Where $D$ is set of training examples