BSc Computer Science and Mathematics / Course details

The first part of this unit continues the development of Probability theory from Year 1 and provides an important basis for many later courses in Probability and Finance.   

The second part of this course unit provides students with the methodological foundations in model-based statistical learning, in particular likelihood estimation and inference and Bayesian learning. The theoretical and methodological discussions are complemented by practical computer application. 

The first part of the course unit aims to develop a solid foundation in the calculus of probabilities and indicate the relevance and importance of this to tackling real-life problems. 

The second part aims to  

• introduce the foundations of model-based statistical learning, 
• introduce the general principles of likelihood-based inference and testing for general models (i.e. for both discrete and continuous distributions), 
• offer a first overview of Bayesian statistical methods, and 
• demonstrate corresponding computational procedures in R.

On the successful completion of the course, students will be able to:

• calculate marginal distributions of and conditional distributions associated with multivariate random variables 
• describe a range of parametric families to model their probability distribution 
• calculate expectations and conditional expectations 
• evaluate the distribution of functions of random variables 
• use the properties of generating functions to derive moments of distributions, distributions of sums of random variables and limit distributions for sequences of random variables.
• apply model-based approaches in statistical data analysis; 
• derive maximum likelihood estimates and compute corresponding confidence intervals; 
• perform statistical testing from a likelihood perspective; 
• solve standard modelling and inference problems from a Bayesian point of view; 
• use R to apply techniques from the course on actual data.  

Syllabus: 
Semester 1: 
Chapter 1: Random Variables (4 lectures) Definition of Events and their probabilities; definition of Random variables and their distributions; features of Discrete and Continuous random variables; Functions of random variables and mixed random variables. 

Chapter 2: Multivariate random variables (6 lectures) Bivariate Distributions; Independence; Sums of several variables; Conditional distributions; the bivariate transform. 

Chapter 3: Expectation (6 lectures) Expectation of a univariate random variable; Variance and higher Moments; Expectation of a bivariate random variable and conditional expectation; Probability generating functions; Moment generating functions; Sums of random variables using generating functions. 

Chapter 4: Sampling and convergence (6 lectures); The sample mean; Central limit theorem; Chebyshev's Inequality; Poisson Limit Theorem and characteristic functions; Introduction to the multivariate normal distribution. 

Semester 2: 

Section I - Likelihood 

- Entropy foundations: Shannon and differential entropy, cross-entropy, Kullback-Leibler (KL) divergence, expected Fisher information, minimum KL divergence and maximum likelihood. 

- Likelihood-based estimation: Likelihood function, regular models, score function, maximum likelihood estimators (MLE), invariance principle, relationship to ordinary least-squares estimation (OLS), observed Fisher information. 

- Quadratic approximation and normal asymptotics: Quadratic approximation of log-likelihood function and normal distribution, quantifying the uncertainty of MLEs using Fisher information, (squared) Wald statistic, normal confidence intervals, non-regular models. 

-  Likelihood-based inference: Likelihood-based confidence interval, Wilks log-likelihood ratio statistic, likelihood ratio test, generalised likelihood ratio test, optimality properties. 

Section II - Bayes 

- Conditioning and Bayes rule:  Conditional probability, Bayes’ theorem, conditional mean and variance, conditional entropy and chain rules, complete data log-likelihood, observed data log-likelihood, learning unobservable states using Bayes theorem. 

- Principles of Bayesian learning:  Prior and posterior probabilities and densities over parameters, marginal likelihood, sequential updates, summaries of posterior distributions and credible intervals, Bayesian and frequentist interpretation of probability. 

- Standard models: Beta-binomial model (for a proportion), normal-normal model (for the mean), inverse-gamma-normal model (for the variance), properties of Bayesian learning. 

- Bayesian model comparison: Log-marginal likelihood as penalised likelihood, model complexity, Bayes factor, Schwarz approximation and Bayesian Information Criterion (BIC), Bayesian testing using false discovery rate. 

- Choosing priors and optimality properties: default priors, uninformative priors, empirical Bayes, shrinkage estimation, James-Stein estimator, Frequentist properties of Bayesian estimators, optimality of Bayes inference (e.g. Cox theorem). 

This course unit forms the core of Theme C, and as such is delivered over 2 semesters.  Teaching is composed of two hours of lectures per week, and one tutorial class per fortnight.  Teaching materials will be uploaded to Blackboard for reference and review.

Other: one mid-term online timed test and one mid-term written test, 40-50 mins each.

Marked scripts within a week of the in-class test and automatically given feedback following the online test. Generic feedback made available after marks are released.

Semester 1: 

Mood, A. M., Graybill, F. A. and Boes, D. C., Introduction to the Theory of Statistics, 3rd edition, McGraw-Hill 1974 

S. Ross, A First Course in Probability, 4th edition, Macmillan. 

D. Stirzaker, Elementary Probability, Cambridge University Press. Available electronically 

Neil A. Weiss, A Course in Probability, Pearson. 

Semester 2: 

Strimmer, K. 2024. MATH27720 Part 2 lecture notes. (Essential) 

Held, L, and Bove, D.S. 2020.  Applied Statistical Inference: Likelihood and Bayes (2nd edition). Springer (Recommended)