Multivariable Differentiation

The heart of the multivariable foundations (analysis spine). The right notion of derivative in several variables is not the collection of partial derivatives but the total (Fréchet) derivative — the best linear approximation, encoded by the Jacobian matrix. Getting this right makes the chain rule a statement about matrix multiplication and sets up the tangent-space picture that manifolds (topology-manifolds.md §3) take for granted.

References: Rudin, Principles of Mathematical Analysis, ch. 9; Spivak, Calculus on Manifolds, ch. 2.

1. Partial and Directional Derivatives

For $f:U\to \mathbb{R}$, $U\subseteq {\mathbb{R}}^n$ open, the partial derivative in the $j$-th coordinate is

$$\frac{\partial f}{\partial x^j}(a)=\underset{t\to 0}{\lim }\frac{f(a+t{e}_j)-f(a)}{t},$$

and more generally the directional derivative along a unit vector $v$ is $D_{v}f(a)={\lim }_{t\to 0}\frac{f(a+tv)-f(a)}{t}$. Warning: existence of all partials (even all directional derivatives) does not imply continuity. The function $f(x,y)=xy/(x^2+y^2)$ (with $f(0,0)=0$) has both partials at the origin yet is discontinuous there. Partials alone are too weak; we need the total derivative.

2. The Total (Fréchet) Derivative

A map $f:U\to {\mathbb{R}}^m$ is (totally) differentiable at $a$ if there is a linear map $Df(a):{\mathbb{R}}^n\to {\mathbb{R}}^m$ with

$$\underset{h\to 0}{\lim }\frac{\Vert f(a+h)-f(a)-Df(a)h\Vert }{\Vert h\Vert }=0.$$

$Df(a)$ is the derivative (or differential) of $f$ at $a$ — the unique best linear approximation, exactly the multivariable form of the one-variable statement $f(a+h)=f(a)+f^{\prime }(a)h+o(h)$ (differentiation.md §1). Differentiability at $a$ implies continuity at $a$.

2.1 The Jacobian matrix

In standard coordinates $Df(a)$ is represented by the Jacobian matrix of partial derivatives,

$$Df(a)={\left[\frac{\partial f^i}{\partial x^j}(a)\right]}_{\begin{array}{c}1\le i\le m\\ 1\le j\le n\end{array}}\in {\mathbb{R}}^{m\times n}.$$

If $f$ is differentiable then all partials exist and $Df(a)$ is this matrix; the converse needs a regularity hypothesis:

Sufficient condition ($C^1\Rightarrow$ differentiable). If all partial derivatives exist and are continuous on $U$, then $f$ is differentiable on $U$ (such $f$ is called continuously differentiable, $f\in C^1$).

So in practice "the partials are continuous" is the checkable certificate of differentiability.

2.2 Gradient

For scalar $f:U\to \mathbb{R}$, $Df(a)$ is a row vector; its transpose is the gradient

$$\nabla f(a)=(\frac{\partial f}{\partial x^1},\dots ,\frac{\partial f}{\partial x^n})(a),D_{v}f(a)=\nabla f(a)\cdot v.$$

The gradient points in the direction of steepest ascent and is orthogonal to level sets — the geometric facts behind Lagrange multipliers and the vector-calculus operator $\nabla$ (vector-calculus.md §2).

3. The Chain Rule

Chain Rule. If $g:U\to {\mathbb{R}}^m$ is differentiable at $a$ and $f:V\to {\mathbb{R}}^p$ is differentiable at $g(a)$, then $f\circ g$ is differentiable at $a$ and $$D(f\circ g)(a)=Df(g(a))\circ Dg(a),$$ i.e. the Jacobian of a composite is the product of Jacobian matrices.

Proof idea. Compose the two best-linear-approximations; the cross error terms are $o(\Vert h\Vert )$ because $Df$ is bounded (a linear map on ${\mathbb{R}}^n$ is Lipschitz). $\square$

This is the cleanest justification for the linear-algebra packaging of the derivative: the messy multivariable chain rule $\frac{\partial }{\partial x^k}(f\circ g)={\sum }_j\frac{\partial f}{\partial y^j}\frac{\partial g^j}{\partial x^k}$ is just the $(i,k)$ entry of a matrix product.

4. Higher Derivatives and the Symmetry of Mixed Partials

Second partials ${\partial }^{2}f/\partial x^i\partial x^j$ assemble into the Hessian matrix $Hf(a)=[{\partial }^{2}f/\partial x^i\partial x^j]$.

Clairaut–Schwarz Theorem. If the second partials of $f$ are continuous on $U$ (i.e. $f\in C^2$), then mixed partials commute: $$\frac{{\partial }^{2}f}{\partial x^i\partial x^j}=\frac{{\partial }^{2}f}{\partial x^j\partial x^i}.$$

Hence the Hessian is symmetric. This symmetry is exactly the algebraic fact $d^2=0$ in disguise — it is what makes the exterior derivative well-defined and what gives $\mathrm{curl}\mathrm{grad}=0$ and $\mathrm{div}\mathrm{curl}=0$ (differential-forms.md §4).

5. Taylor's Theorem in Several Variables

For $f\in C^{k+1}$ near $a$,

$$f(a+h)=\sum _{|\alpha |\le k}\frac{D^{\alpha }f(a)}{\alpha !}{h}^{\alpha }+R_k(h),R_k(h)=O(\Vert h{\Vert }^{k+1}),$$

in multi-index notation. The second-order expansion

$$f(a+h)=f(a)+\nabla f(a)\cdot h+\frac{1}{2}{h}^{\top }Hf(a)h+o(\Vert h{\Vert }^2)$$

drives the second-derivative test: at a critical point ($\nabla f=0$), a positive-definite Hessian gives a local min, negative-definite a local max, indefinite a saddle. This is the one-variable Taylor theorem (differentiation.md §6) propagated along rays.

6. The Mean Value Inequality

The exact one-variable MVT ($f(b)-f(a)=f^{\prime }(\xi )(b-a)$) has no vector-valued analogue (there may be no single $\xi$ for all components). The usable replacement is an inequality:

Mean Value Inequality. If $f:U\to {\mathbb{R}}^m$ is $C^1$ and the segment $[a,a+h]\subseteq U$, then $\Vert f(a+h)-f(a)\Vert \le ({\sup }_{t\in [0,1]}>\Vert Df(a+th)\Vert )\Vert h\Vert$.

This bound — a derivative controlling an increment — is what powers the convergence estimates in the contraction-mapping proof of the inverse function theorem (inverse-implicit.md).

7. Connection to Tangent Spaces

The derivative-as-linear-map is the flat model of the manifold notion of differential. For a smooth map $F:M\to N$ between manifolds, the pushforward $d{F}_p:T_{p}M\to T_{F(p)}N$ is, in charts, precisely the Jacobian $D(\psi \circ F\circ {\varphi }^{-1})$ of this page (topology-manifolds.md §3). Everything here is the local coordinate computation that the intrinsic manifold language packages coordinate-free.

8. Where this page is used

• Total derivative + contraction estimate $\to$ inverse and implicit function theorems (inverse-implicit.md).
• Jacobian determinant $\to$ change of variables in multiple-integrals.md.
• Gradient, symmetry of second partials $\to$ grad/div/curl and the identities $\mathrm{curl}\mathrm{grad}=0$, $\mathrm{div}\mathrm{curl}=0$ (vector-calculus.md, differential-forms.md).
• Pushforward / tangent map $\to$ integration of forms on manifolds (stokes-general.md).

Next: Inverse and Implicit Function Theorems.