Learnings from https://www.youtube.com/watch?v=SmZmBKc7Lrs

Problem to solve - Curve Fitting Problem

With back propagation, we can solve problems such as finding the best fit curve for a set of data points.

For example, given this:

We can find out what are the values of k0, k1… k5 through continuous tuning.

To tune, we need backpropagation.

Evaluation - How do we know what is the best curve?

To know what defines what the best curve is, we need a loss function.

A common loss function is the Squared Distance.

How to get the best curve?

We have to keep measuring the loss from the loss function, and see if the loss is decreasing when we tune the value of each k0 to k5.

We can do it randomly via random perturbation. (Random tuning)

However, how can we be more efficient in tuning each values?

Derivatives - the more efficient method

Knowing the derivative at each point of the loss function allows us to know whether to decrease or increase the value of k.

For example, if the loss is decreasing (negative derivative) at a point, we can continue to increase the value of k.

Multiple dimensions - Gradient Vector

The same concept applies to multiple dimension. With this gradient vector, we can know how to reach the minimum point (minimum loss) via gradient descent.

Chain Rule

We can obtain derivatives of any arbitrary complicated functions.

How do we make use of these knowledge to find the best curve?

If the curve function is composed of a chain of differential functions (or multilayered), it is possible to know how to turn the knobs in such a way that we can minimize the loss. (via chain rule)

With each forward pass, we know the loss.

With each backward pass, we know the rate of change of loss for each variable.

And then, we nudge the knobs. (And repeat)

Hence, this is what back propagation is about.

Back Propagation