Notes from Wednesday 10/09.

Structure: How does a neural network recognize handwritten digits?.

His demonstration is a basic one ("multilayer perceptron").

• a "neuron" takes in the inputs of the previous function, and outputs a single value (its "activation")
• the first layer is a list of neurons that hold an activation number, which corresponds to how darkly shaded each pixel of the input image is
• the last layer is a list of our possible digits, with its neurons containing activations corresponding to the likelihood of that digit being equal to the input image

What do the middle layers do?

→ the first breaks it down into small, recognizable sub-components, and each remaining layer builds them back up

*similar process for parsing speech: neural networks take raw audio, break them down into basic sounds, and then build them up into recognizable syllables and then words

How does one layer influence the next?

→ Each individual neuron in one layer is influenced by all the neurons in the previous one by an equation that combines their activation numbers with weights and adds a bias, then squishes that value between 0 - 1 (using a sigmoid function).

*the bias is meant to de-activate false positives; e.g. we only want to activate the next neuron if b > 10

$$ a_0^0 = \sigma (w_1a_1+w_2a_2+...+w_na_n - b) $$

→ We commonly express this formula using matrices

$$ \begin{bmatrix} w_{0,0} & w_{0,1} & \dots & w_{0,n} \\ w_{1,0} & w_{1,1} & \dots & w_{2,n} \\ \vdots & \vdots & \ddots & \vdots \\ w_{k,0} & w_{k,1} & \dots & w_{k,n} \end{bmatrix} \begin{bmatrix} a_0^0 \\ a_1^0 \\ \vdots \\ a_n^0 \end{bmatrix} + \begin{bmatrix} b_0 \\ b_1 \\ \vdots \\ b_n \end{bmatrix} = \begin{bmatrix} a_0^1 \\ a_1^1 \\ \vdots \\ a_n^1 \end{bmatrix} $$

Learning

To increase accuracy, we train the network with sample data, and then test it on new data to see its accuracy.

• We use the test data evaluate the network's accuracy by defining a "cost function," which compares the expected result to the actual.

  The cost function adds the squares of the differences between the expected and actual for each result neuron (the neurons of the last layer). A small sum is good; a large sum is bad.

  To measure the network's efficacy as a whole, we take the average of all costs.

• But we also want to tell the network how to improve, not just display how good or bad it is. For this we use calculus to improve the values of our weights.

  We can view the cost function as taking one input (a weight) and spitting out one output (a cost value). Then to find the minimum (the lowest cost value), we need to find the minimum of the cost function.

  But we do this by "stepping" in the direction of the minimum cost (by evaluating the current slope). We have to do this because the cost function is too complicated to find its global minimum directly.

  This entire process is called backpropagation, and finding the local minimums of the cost function is called gradient descent.