Hessian Matrix

The Hessian Matrix is a square matrix of second ordered partial derivatives of a scalar function. It is of immense use in linear algebra as well as for determining points of local maxima or minima.

The above Hessian is of the the function $f : \mathbb{R}''\to\mathbb{R}$ where all second order partial derivatives of $f$ exist and are continuous throughout its domain & the function is $f(x_1,x_2,x_3,\cdots,x_n)$

The Hessian of a function is denoted by $\Delta^2f(x,y)$ where $f$ is a twice differentiable function & if $(x_0,y_0)$ is one of its stationary points then :

• If $\Delta^2f(x_0,y_0)>0$ i.e positive definite , $(x_0,y_0)$ is a point of local minimum.
• If $\Delta^2f(x_0,y_0)<0$ , i.e. negative definite , $(x_0,y_0)$ is a point of local maximum.
• If $\Delta^2f(x_0,y_0)$ is neither positive nor negative i.e. Indefinite , $(x_0,y_0)$ is a saddle point

Usually Hessian in two variables are easy and interesting to look for. A function $f:\mathbb{R}\to\mathbb{R}$ whose second order partial derivatives are well defined in its domain so we can have the Hessian matrix of $f$ .

Note that the Hessian matrix here is always symmetric.

Let the function $f(x,y)= x^2+y^2$ satisfies that its second order partial derivatives exist & they're continuous throughout the Domain .

$\large \begin{cases} \frac{\partial{f}^2}{\partial{x^2}}=2 \\ \frac{\partial{f}^2}{\partial{y}\partial{x}} = 0 \\ \frac{\partial{f}^2}{\partial{x}\partial{y}}=0 \\ \frac{\partial{f}^2}{\partial{y^2}} = 2\end{cases}$

Then its Hessian is given & denoted by :

$\large \Delta^2f(x,y) = \begin{pmatrix} \frac{\partial{f}^2}{\partial{x^2}} & \frac{\partial{f}^2}{\partial{y}\partial{x}} \\ \frac{\partial{f}^2}{\partial{x}\partial{y}} & \frac{\partial{f}^2}{\partial{y^2}} \end{pmatrix}$

$\large \Delta^2f(x,y) = \begin{pmatrix} 2&0 \\ 0&2 \end{pmatrix}$

The following example will demonstrate the facts clearly & explain its uses.

Suppose a function is defined by $f(x,y)=x^4-32x^2+y^4-18y^2$ . Find the maximum & minimum value of the function if it exists. Justify your answer.

We take the double derivatives of the function as follows :

$\large \begin{cases} \frac{\partial{f}^2}{\partial{x^2}}=12x^2-64 \\ \frac{\partial{f}^2}{\partial{y}\partial{x}} = 0 \\ \frac{\partial{f}^2}{\partial{x}\partial{y}}=0 \\ \frac{\partial{f}^2}{\partial{y^2}} = 12y^2-36\end{cases}$

The Hessian is defined by :

$\large \Delta^2f(x,y) = \begin{pmatrix} 12x^2-64&0 \\ 0&12y^2-36 \end{pmatrix}$

We solve for the Stationary points of the function $f(x,y)$ by equating its partial derivatives $\frac{\partial{f}}{\partial{x}}$ & $\frac{\partial{f}}{\partial{y}}$ to zero.

$\begin{cases} 4x(x^2-16)=0\implies x=\pm4,0 \\ 4y(y^2-9)=0\implies y=\pm3,0\end{cases}$

The possible pairings gives us the critical points $(0,0),(\pm4,\pm3),(\pm4,0),(0,\pm3)$ .

Now as the Hessian consists of even functions which reduces a lot of effort. we only need to check for the pairs $(4,3),(0,0),(4,0),(0,3)$

For a brief knowledge of Definite & indefinite matrices study these first.

Now we check the Hessian at different stationary points as follows :

$\large \Delta^2f(0,0) = \begin{pmatrix} -64 &0 \\ 0 & -36\end{pmatrix}$

This is negative definite making it a local maximum of the function .

$\large \Delta^2f(\pm4,0) = \begin{pmatrix} 128 &0 \\ 0 & -36\end{pmatrix}$

It's indefinite thus ruled out.

$\large \Delta^2f(0,\pm3) = \begin{pmatrix} -64 &0 \\ 0 & 72\end{pmatrix}$

It's also indefinite

$\large \Delta^2f(\pm4,\pm3) = \begin{pmatrix} 128 &0 \\ 0 & 72\end{pmatrix}$

It's positive definite matrix and thus it's the local minimum of the function.

So $f(0,0)\ge f(x,y)\ge f(\pm4,\pm3)\implies -337\le f(x,y)\le0$

Thus we have successfully bounded the above function and its point of local minimum is $(\pm4,\pm3)$ & point of local maximum is $(0,0)$