ELEC 321

Random Process

Updated 2017-10-31

Also known as the stochastic processes. It is the outcome of the random experiment as a function of time or space, etc. They are random variables indexed by the time or space variable.

If the index is countable set, then the random process is discrete-time. Else if the index is continuous, then the random process is continuous-time.

Characterization and Statistics

Discrete-time (DT) Random Processes

Definition: A family of random variables , indexed by discrete time and outcomes .

Each outcome in the sample space yields a sample path

We denote this random process as .

Specifying DT Random Processes

We have PMF and CDF to describe random variables, but how do we describe random processes?

We characterize a DT random process by joint probabilities for vectors of samples of the process.

equation on slide 5

Independent and Identically Distributed Processes

A DT random process is IID if the two properties hold:

  1. Independent (no memory)

  2. Identically distributed (time-invariant distribution)

Because of the two properties, the joint PMF becomes the product:

Example: flipping a coin

Flipping heads or tails follows the Bernulli random variable.

The joint probability of flipping the sequence {H, T, T, H} is:

Example: counting arrivals

The random variable follows the Poisson distribution where .

Example: counting arrivals with time

If random variable . In particular, for

This would not be an IID process since the distribution changes with time, and thus not identical.

The solution is that and .

Non-IID DT Random Processes

If two random variables are statistically independent, then it follows that they are uncorrelated. By the same logic, to show that if the random variables are dependent, we could show that the two random variables are correlated.

An example of non-IID random process are Markov Chains. In a Markov chain, the current only depends on the one previous, .

Statistics of Random Processes

Consider DT random process .


The mean of a random process is a deterministic time-varying signal:

For IID processes, .


Deterministic signal in two variables

This is the similarity of a signal with itself at different points in time.

For IID processes:

So if is the same as (same time), then the auto-covariance is the variance of , which is constant (since its not changing in time as the distributions are identical). Otherwise, it is 0 since independent implies no correlation.


This is the similarity of a signal with another signal at a different time.

Auto-Correlation Function

ACF is a deterministic signal in two variables where

If (or in continuous case), then is the average power output of the signal.

Discrete-Time Continuous Valued Process

should be a summation???

Continuous-Time Continuous Valued Process
Correlation Coefficient

photo on Poisson process

Random Process an example of continuous time random processes based on IID processes

Stationary and Ergodic Processes

notes from slides

Example 1

Let be an IID process, and let , is stationary?

Note that IID processes are strict-sense stationary. Because of central limit theorem, as we have more random variables ( increases), will have a normal distribution, and the parameters for normal distribution are mean and variance. Observe the mean:

We see that the mean increases as the we add more , thus is not wide-sense stationary. Thus it is also not strict-sense stationary.

Example 2

Let consist of two interleaved sequences of independent random variables.

This is not strict-sense stationary.

But it is wide-sense stationary, since . Since the random variables are independent, the covariance is , and Kronecker delta function is time invariant.

If a process is stationary, it may further be Ergodic.


Example 1

Spectral Density

Frequency Response

Example 1: white process

Consider random process where its ACF is (scaled delta function).

Taking the Fourier transform of the ACF, we get , since the Fourier Transform of the Kronecker delta function is 1. The ACF in frequency domain is a constant (spectrally white - all frequency contributes the same power).

is a white process.

Example 2

If the random process has Gaussian distribution: , then is a white Gaussian process.

Processes and Linear Time-invariant Systems

White process, after LTI sys, . The is referred to as “Coloring”

Back to time domain, .

Application: MMSE Linear Approximation

Preliminaries: estimate a random variable based on observation of such that the mean-square error (MSE) is minimized.

where is the prediction of actual . We want the mean squared error to be minimum:

  1. Given no observation, then the argument that will minimize the error is given as

    Do the derivative to find the minimum

  2. Linear estimator (we have a scale and a shift )

    So we need to minimize using the argument and :

    Again, take derivative to find minimum:

    Turns out,