ELEC 321

# Multivariate Normal

Updated 2017-12-07

## Standard Multivariate Normal

Recall that a standard normal random variable has a expected value / mean of 0 and a variance of 1. Then a random vector of standard normal is denoted as $\mathbf Z \sim N(\mathbf 0, I)$. Where $Z$ is the random vector with a size of $N$, $\mathbf 0$ is a vector of zeroes, and $I$ is a $N\times N$ identity matrix.

### Density

Suppose there are independent standard normal random variables $Z_1, Z_2, \dots, Z_n$ then their joint density is the product:

$f(z_1, z_2,\dots,z_N)=\prod_{i=1}^N\varphi(z_i)$

Recall that

$\varphi (z)= \frac{1}{\sqrt{2\pi}}e^{-\frac{1}{2}z^2}$

Then the joint density can be simplified to

$f(\mathbf z)=\frac{1}{(\sqrt{2\pi})^N}e^{-\frac{1}{2}\mathbf{z'z}}$

Where $\mathbf z$ is a vector that contains $z_1,\dots,z_N$, and $\mathbf {z'z}$ is the dot product of itself.

### Mean

The expected value of the random vector composed of standard multivariate normal is a vector of zeroes:

$\boldsymbol \mu= \mathbb E\{\mathbf Z\}=\mathbf 0$

### Covariance

For $N$ random variables, the covariance matrix is an $N\times N$ identity matrix.

## General Multivariate Normal

Given that $\mathbf Z$ is the random vector of standard multivariate normal; any multivariate normal random vector can take the form:

$\mathbf X=A\mathbf Z + \mathbf b$

Where $\mathbf X$ is the random vector comprised of random variables with general normal distribution; vector $\mathbf X$ has a length of $N$ . $A$ is an $N\times N$ invertible matrix. And $\mathbf b$ is a vector of constants.

Because $A$ is invertible, then

$\mathbf Z=A^{-1}(\mathbf X-\mathbf b)$

### Mean

The mean of a generate multivariate normal is $\mathbf b$ since the mean for the standard multivariate normal is $\mathbf 0$:

$\boldsymbol \mu = \mathbb E\{\mathbf X\}=\mathbb E\{A\mathbf Z+\mathbf b\}=\mathbf b$

### Covariance

The covariance is $AA'$ where $A'$ is the transpose of $A$:

$\Sigma=\text{Cov}(\mathbf X)=\text{Cov}(A\mathbf Z+\mathbf b)=\text{Cov}(A\mathbf Z)=A\text{Cov}(\mathbf Z)A'=AA'$

### Jacobian

Notice that $\mathbf X=A\mathbf Z + \mathbf b$ is in fact a transformation, therefore its corresponding Jacobian is simply

$J=\vert \det(A^{-1})\vert =\frac{1}{\vert \det(A)\vert }$

### Density

Now that we know the Jacobian, it follows that the density is given by

$f_\mathbf X(\mathbf x)=\frac{1}{\vert \det(A)\vert }f_{\mathbf Z}(A^{-1}(\mathbf x-\mathbf b))$

Plugging in $f(\mathbf z)=\frac{1}{(\sqrt{2\pi})^N}e^{-\frac{1}{2}\mathbf{z'z}}$, and we obtain

$\frac{1}{\vert \det(A)\vert } \frac{1}{(\sqrt{2\pi})^N}e^{-\frac{1}{2}\mathbf{(A^{-1}(\mathbf x-\mathbf b))'(A^{-1}(\mathbf x-\mathbf b))}}$

Note that the covariance matrix $\Sigma=AA'$, which also implies $\Sigma^{-1}=(A^{-1})'A^{-1}$.

Also note that the determinant of the covariance matrix is $\det(\Sigma)=\det(AA')$, working it out we see that $\sqrt{\det(\Sigma)}=\vert \det(A)\vert$.

Plugging these equations in, the above density simplify down to

$f_\mathbf X(\mathbf x)=\frac{1}{\sqrt{(2\pi)^N\Sigma}}e^{-\frac{1}{2}(\mathbf x-\mathbf b)'\Sigma^{-1}(\mathbf x-\mathbf b)}$

## Properties of Multivariate Normal

1. Linear transformation of normal vectors results in normal vectors

Suppose we have a vector $\mathbf X\sim N(\boldsymbol \mu, \Sigma)$, and a matrix $C$ is full rank. then let $\mathbf Y=C\mathbf X+\mathbf d$. The mean of $\mathbf Y$ is $C\boldsymbol \mu+\mathbf d$; the variance is $C\Sigma C'$.

2. Marginal distributions are normal

Suppose we have a random vector that has size 2:

$\mathbf X=\begin{bmatrix} X_1\\ X_2\\ \end{bmatrix} \sim N\left( \begin{bmatrix} \mu_1\\ \mu_2\\ \end{bmatrix} , \begin{bmatrix} \Sigma_{11} & \Sigma_{12}\\ \Sigma_{21} & \Sigma_{22}\\ \end{bmatrix} \right)$

Then we get

$X_1\sim N(\mu_1,\Sigma_{11}),\qquad X_2\sim N(\mu_2,\Sigma_{22})$

3. Conditional distributions are normal

Using the previous case, suppose we have $x_2$ as a realization for $X_2$, then the conditional on $X_1$ is

$X_1\vert (X_2=x_2)\sim N(\mu_{1\vert 2}, \Sigma_{1\vert 2})$

Where $\mu_{1\vert 2}=\mu_1+\frac{\Sigma_{12}}{\Sigma_{22}}(x_2-\mu_2)$ and $\Sigma_{1\vert 2}=\Sigma_{11}-\frac{\Sigma_{12}\Sigma_{21}}{\Sigma_{22}}$.