Learning the parameters $\boldsymbol{\theta}$ for a parametric model is typically done by solving an optimization problem of the form

for some objective function $J$. In particular, this is the case when training neural networks. One way to solve this type of problem is by the gradient descent method. Gradient descent can be applied to optimization problems of this type as long as it is possible to compute gradients of $J$ with respect to the parameters $\boldsymbol{\theta}$. Recall that gradients, denoted $\nabla$, is the analog of the derivative to functions of multiple variables (For example the function $J$ over the vector $\boldsymbol{\theta}$). The gradient is a vector where each element is a partial derivative. For an objective function $J$, the gradient with respect to a $d$-dimensional parameter vector $\boldsymbol{\theta}$ is:

Just as the derivative of a function $f$ describes how $f(x)$ varies as $x$ changes, the gradient describes how a function $J(\boldsymbol{\theta})$ varies as each element in $\boldsymbol{\theta}$ changes.

A particularly useful property is that the gradient describes the direction in which $J$ increases most rapidly. Note that this is a direction in the $d$-dimensional parameter space. In the same way the negative gradient, $-\nabla_{\boldsymbol{\theta}} J(\boldsymbol{\theta})$, describes the direction in which $J$ decreases the fastest.

Consider for example the simple function $J(\boldsymbol{\theta}) = J(\theta_1, \theta_2) = \theta_1 + \theta_2$. The parameter vector $\boldsymbol{\theta}$ contains only $\theta_1$ and $\theta_2$, so the parameter space is 2-dimensional in this case. The figure below shows the value of $J(\boldsymbol{\theta})$ for different values of $\theta_1$ and $\theta_2$. The gradient $\nabla J(\boldsymbol{\theta})$ and its negation $- \nabla J(\boldsymbol{\theta})$ are also shown as arrows, evaluated in the point $\boldsymbol{\theta} = [0, 0]^\top$.

From the figure we see clearly that if we want to decrease the value of $J(\boldsymbol{\theta})$ as much as possible, we should change $\boldsymbol{\theta}$ in the direction of $- \nabla J(\boldsymbol{\theta})$. In gradient descent we will use this to find values of $\boldsymbol{\theta}$ that minimize the function $J$.

The high-level idea behind gradient descent is to iteratively update $\boldsymbol{\theta}$ in such a way that $J(\boldsymbol{\theta})$ decreases. If we think of $\boldsymbol{\theta}$ as a point in some $d$-dimensional space, this can be done by taking a small step in the direction of the negative gradient. If we then keep on taking such steps, the value of $J(\boldsymbol{\theta})$ should decrease until we get to a minimum. This idea is formalized in the gradient descent update rule:

The constant $\gamma$, often referred to as learning rate, is the length of each small step in the parameter space. Many methods exists for choosing $\gamma$ in more or less principled ways, but often this is left as a hyperparameter to be tuned by the machine learning practitioner.

Let us look at an example of gradient descent optimization in the very simple case of a one-dimensional $\theta$. In this case the gradient of $J$ is simply the normal derivative. For this example we want to minimize

which has the gradient

The objective function $J$ is also plotted below:

We initialize $\theta^{(0)} = 2$ and run the gradient descent process with learning rate $\gamma=0.1$. The initial value of the objective function is

and the gradient at this point is

We take one gradient descent step by computing $\theta^{(1)}$ using the gradient descent update rule:

This first gradient descent step is illustrated in the figure below:

We can compute the value of the objective function at this new $\theta^{(1)}$: $J(\theta^{(1)}) = J(1.6) = 1.72$. As we expected, the value has decreased after the gradient descent step. If we take another step in the same way we will reach $\theta^{(2)} = 1.36$ and an even lower value for $J$, as illustrated in the following figure. If you want more practice with the algorithm, perform the computations for more gradient descent steps manually.

As this step process is iterated, the value of $\theta$ approaches the minimum at $\theta = 1$. Note that even though we always use the same learning rate $\gamma$, the lengths of the steps vary. This is because the actual step length is also affected by the magnitude of the gradient itself, which is different for different $\theta$. Already after 5 steps the value of $\theta$ is fairly close to the minimum:

In this simple example we could have computed the optimal value of $\theta$ analytically, making gradient descent optimization unnecessary. It does however serve as a good example of the iterative gradient descent process. For most parametric machine learning models $J$ is far more complicated than in this example and $\theta$ is commonly high-dimensional. In these cases gradient-based optimization techniques like gradient descent are usually the best option for finding a minimum of the objective function.

One question worth asking is whether the gradient descent process described above always will find the global minimum $\hat{\boldsymbol{\theta}}$? It turns out that this is not generally the case. Only if the objective function $J$ is convex and $\gamma$ is chosen to fulfill some specific criteria, gradient descent can be guaranteed to find the global minimum. Many optimization problems encountered in machine learning are non-convex, an this is true in particular when training neural networks. Should we then give up on gradient descent as an optimization method? No, this would be a premature conclusion — as it turns out, in many practical optimization problems it is enough to find a $\boldsymbol{\theta}$ that is a local minimum or even just close to a local minimum. In particular this is the case when training neural networks. This means that gradient descent is indeed a very useful method in practice, even though we do not have any guarantees of finding the optimal solution.

The description of gradient descent given here is the most basic form of the algorithm. There are a number of practical aspects to using gradient descent that are worth knowing about. When working with very large amounts of data it might not be computationally feasible (nor necessary) to compute the exact gradient at each iteration. It is then common to approximate the gradient using only a subset of the data (referred to as a mini-batch). This modified version of the algorithm is called Stochastic Gradient Descent (SGD). Mini-batching is more or less always used when training neural networks. SGD can be extended further by incorporating clever schemes for automatically adjusting the learning-rate $\gamma$ throughout the optimization process. These different extensions are often referred to as optimizers. Some common optimizers used for training neural networks are Adam and RMSprop. You can read more about these in chapter 5.5 of the course book.