Overview: Markov chains in discrete time

In this third project we introduce the Markov property and thus begin our study of random, time-dependent processes. In this part of the course we assume that we are monitoring our random process at particular points in time rather than assuming we are monitoring the random process continuously. Furthermore, we assume that the random variable whose value we are monitoring in time can only take integer values. To describe these processes we introduce the notion of a transition graph and a transition matrix. You will learn how to write programs to simulate Markov chains and will learn how quantitative predictions can be made for these random processes by examining the limiting behavior of the chains.


  • You should be able to state what it means when we say that a time series of random variables has the Markov property
  • You should be able to interpret transition graphs and transition probability matrices and you should also be able to use the Chapman-Kolmogorov relation to calculate the $n$-step transition probability matrix from the 1-step transition probability matrix.
  • You should be able to discuss the limiting behavior of Markov chains. This means you should be able to calculate limiting stationary distributions, hitting times and hitting probabilities.
  • You should be able to write a computer program to simulate a random walk in one dimension.

Contact Details

School of Mathematics and Physics,
Queen's University Belfast,

Email: g.tribello@qub.ac.uk
Website: mywebsite