# Multivariate Normal

Updated 2017-12-07

## Standard Multivariate Normal

Recall that a standard normal random variable has a expected value / mean of 0 and a variance of 1. Then a random vector of standard normal is denoted as $$\mathbf Z \sim N(\mathbf 0, I)$$. Where $$Z$$ is the random vector with a size of $$N$$, $$\mathbf 0$$ is a vector of zeroes, and $$I$$ is a $$N\times N$$ identity matrix.

### Density

Suppose there are independent standard normal random variables $$Z_1, Z_2, \dots, Z_n$$ then their joint density is the product:

$f(z_1, z_2,\dots,z_N)=\prod_{i=1}^N\varphi(z_i)$

Recall that

$\varphi (z)= \frac{1}{\sqrt{2\pi}}e^{-\frac{1}{2}z^2}$

Then the joint density can be simplified to

$f(\mathbf z)=\frac{1}{(\sqrt{2\pi})^N}e^{-\frac{1}{2}\mathbf{z'z}}$

Where $$\mathbf z$$ is a vector that contains $$z_1,\dots,z_N$$, and $$\mathbf {z'z}$$ is the dot product of itself.

### Mean

The expected value of the random vector composed of standard multivariate normal is a vector of zeroes:

$\boldsymbol \mu= \mathbb E\{\mathbf Z\}=\mathbf 0$

### Covariance

For $$N$$ random variables, the covariance matrix is an $$N\times N$$ identity matrix.

## General Multivariate Normal

Given that $$\mathbf Z$$ is the random vector of standard multivariate normal; any multivariate normal random vector can take the form:

$\mathbf X=A\mathbf Z + \mathbf b$

Where $$\mathbf X$$ is the random vector comprised of random variables with general normal distribution; vector $$\mathbf X$$ has a length of $$N$$ . $$A$$ is an $$N\times N$$ invertible matrix. And $$\mathbf b$$ is a vector of constants.

Because $$A$$ is invertible, then

$\mathbf Z=A^{-1}(\mathbf X-\mathbf b)$

### Mean

The mean of a generate multivariate normal is $$\mathbf b$$ since the mean for the standard multivariate normal is $$\mathbf 0$$:

$\boldsymbol \mu = \mathbb E\{\mathbf X\}=\mathbb E\{A\mathbf Z+\mathbf b\}=\mathbf b$

### Covariance

The covariance is $$AA'$$ where $$A'$$ is the transpose of $$A$$:

$\Sigma=\text{Cov}(\mathbf X)=\text{Cov}(A\mathbf Z+\mathbf b)=\text{Cov}(A\mathbf Z)=A\text{Cov}(\mathbf Z)A'=AA'$

### Jacobian

Notice that $$\mathbf X=A\mathbf Z + \mathbf b$$ is in fact a transformation, therefore its corresponding Jacobian is simply

$J=\vert \det(A^{-1})\vert =\frac{1}{\vert \det(A)\vert }$

### Density

Now that we know the Jacobian, it follows that the density is given by

$f_\mathbf X(\mathbf x)=\frac{1}{\vert \det(A)\vert }f_{\mathbf Z}(A^{-1}(\mathbf x-\mathbf b))$

Plugging in $$f(\mathbf z)=\frac{1}{(\sqrt{2\pi})^N}e^{-\frac{1}{2}\mathbf{z'z}}$$, and we obtain

$\frac{1}{\vert \det(A)\vert } \frac{1}{(\sqrt{2\pi})^N}e^{-\frac{1}{2}\mathbf{(A^{-1}(\mathbf x-\mathbf b))'(A^{-1}(\mathbf x-\mathbf b))}}$

Note that the covariance matrix $$\Sigma=AA'$$, which also implies $$\Sigma^{-1}=(A^{-1})'A^{-1}$$.

Also note that the determinant of the covariance matrix is $$\det(\Sigma)=\det(AA')$$, working it out we see that $$\sqrt{\det(\Sigma)}=\vert \det(A)\vert$$.

Plugging these equations in, the above density simplify down to

$f_\mathbf X(\mathbf x)=\frac{1}{\sqrt{(2\pi)^N\Sigma}}e^{-\frac{1}{2}(\mathbf x-\mathbf b)'\Sigma^{-1}(\mathbf x-\mathbf b)}$

## Properties of Multivariate Normal

1. Linear transformation of normal vectors results in normal vectors

Suppose we have a vector $$\mathbf X\sim N(\boldsymbol \mu, \Sigma)$$, and a matrix $$C$$ is full rank. then let $$\mathbf Y=C\mathbf X+\mathbf d$$. The mean of $$\mathbf Y$$ is $$C\boldsymbol \mu+\mathbf d$$; the variance is $$C\Sigma C'$$.

2. Marginal distributions are normal

Suppose we have a random vector that has size 2:

$\mathbf X=\begin{bmatrix} X_1\\ X_2\\ \end{bmatrix} \sim N\left( \begin{bmatrix} \mu_1\\ \mu_2\\ \end{bmatrix} , \begin{bmatrix} \Sigma_{11} & \Sigma_{12}\\ \Sigma_{21} & \Sigma_{22}\\ \end{bmatrix} \right)$

Then we get

$X_1\sim N(\mu_1,\Sigma_{11}),\qquad X_2\sim N(\mu_2,\Sigma_{22})$

3. Conditional distributions are normal

Using the previous case, suppose we have $$x_2$$ as a realization for $$X_2$$, then the conditional on $$X_1$$ is

$X_1\vert (X_2=x_2)\sim N(\mu_{1\vert 2}, \Sigma_{1\vert 2})$

Where $$\mu_{1\vert 2}=\mu_1+\frac{\Sigma_{12}}{\Sigma_{22}}(x_2-\mu_2)$$ and $$\Sigma_{1\vert 2}=\Sigma_{11}-\frac{\Sigma_{12}\Sigma_{21}}{\Sigma_{22}}$$.