BSc Computer Science and Mathematics with Industrial Experience

Year of entry: 2024

Course unit details:
Markov Processes

Course unit fact file
Unit code MATH37011
Credit rating 10
Unit level Level 3
Teaching period(s) Semester 1
Available as a free choice unit? No

Overview

Markov chains are stochastic processes with the special property of "given the present, the future is independent of the past". Many real-life situations can be modelled by such processes and this course is concerned with their mathematical analysis. It by-passes the measure-theoretic considerations necessary for the development of a general theory of stochastic processes.

Pre/co-requisites

Unit title Unit code Requirement type Description
Probability and Statistics 2 MATH27720 Pre-Requisite Compulsory
MATH37011 PRE REQS

Aims

To develop the idea that processes evolving randomly in time can be modelled mathematically in terms of sequences or families of dependent random variables.

Learning outcomes

On completion of this course unit students should be able to:

  • classify the states of a discrete time Markov chain;
  • calculate the stationary and limiting distributions of discrete time Markov chains;
  • state and prove results about recurrence, transience and periodicity of discrete time Markov chains;
  • write down and solve the forward equations for simple birth- death processes;
  • apply continuous time Markov chain theory to the modelling of queues;
  • state and prove results about the limiting behaviour of continuous time Markov chains.

 

Future topics requiring this course unit

The material of this course may be helpful in understanding the 4th year courses on stochastic calculus and Brownian motion.

 

Syllabus

1.Discrete time Markov chains:

Review of necessary probability theory. [2]

Definition of Markov chain. Homogeneity. (1-step) transition probabilities. Transition diagrams. Examples including Ehrenfest diffusion model. The Chapman-Kolmogorov equations. Matrix form. Accessibility. Closed/irreducible sets. Periodicity. Stationary distributions. Positive recurrence, null recurrence, transience. Random walk examples. Convergence to stationary distribution. Discussion of different methods of proof (e.g. Markov's method for finite state space; Doeblin's coupling; renewal type argument). [8]

2.Continuous time Markov chains:

Theoretical treatment at level of Karlin and Taylor (see below). Revision of Poisson process. Pure birth/birth death processes. [8]

3.Applications:

Queues. M/M/1. Queue length and waiting time distribution. M/M/s. Variable arrival and service rates. Machine interference. [4]

Assessment methods

Method Weight
Other 20%
Written exam 80%
  • Coursework: two hours weighting 20%
  • End of semester examination: weighting 80%

Feedback methods

Feedback tutorials will provide an opportunity for students' work to be discussed and provide feedback on their understanding.  Coursework or in-class tests (where applicable) also provide an opportunity for students to receive feedback.  Students can also get feedback on their understanding directly from the lecturer, for example during the lecturer's office hour.

Recommended reading

  • D. R. Stirzaker, Stochastic Processes and Models, Oxford University Press, 2005.
  • A. N. Shiryaev, Probability, Springer-Verlag, 1996.
  • S. Karlin and H. M. Taylor, A First Course in Stochastic Processes, Academic Press, 1975.
  • D. R. Stirzaker, Elementary Probability, Cambridge University Press, 2003.
  • G. R. Grimmett and D. R. Stirzaker, Probability and Random Processes, Oxford Univ. Press, 1992.

Study hours

Scheduled activity hours
Lectures 12
Tutorials 12
Independent study hours
Independent study 76

Teaching staff

Staff member Role
Jose Pedraza Ramirez Unit coordinator

Additional notes

The independent study hours will normally comprise the following. During each week of the taught part of the semester:
 
•         You will normally have approximately 60-75 minutes of video content. Normally you would spend approximately 2-2.5 hrs per week studying this content independently
•         You will normally have exercise or problem sheets, on which you might spend approximately 1.5hrs per week
•         There may be other tasks assigned to you on Blackboard, for example short quizzes or short-answer formative exercises
•         In some weeks you may be preparing coursework or revising for mid-semester tests
 
Together with the timetabled classes, you should be spending approximately 6 hours per week on this course unit.
The remaining independent study time comprises revision for and taking the end-of-semester assessment.
 
The above times are indicative only and may vary depending on the week and the course unit. More information can be found on the course unit’s Blackboard page.

Return to course details