# BSc Computer Science and Mathematics / Course details

Year of entry: 2024

## Course unit details:Probability and Statistics 2

Unit code MATH27720 20 Level 2 Full year No

### Overview

The first part of this unit continues the development of Probability theory from Year 1 and provides an important basis for many later courses in Probability and Finance.

The second part of this course unit provides students with the methodological foundations in model-based statistical learning, in particular likelihood estimation and inference and Bayesian learning. The theoretical and methodological discussions are complemented by practical computer application.

### Pre/co-requisites

Unit title Unit code Requirement type Description
Probability I MATH11711 Pre-Requisite Compulsory
Statistics I MATH11712 Pre-Requisite Compulsory

### Aims

The first part of the course unit aims to develop a solid foundation in the calculus of probabilities and indicate the relevance and importance of this to tackling real-life problems.

The second part aims to

• introduce the foundations of model-based statistical learning,
• introduce the general principles of likelihood-based inference and testing for general models (i.e. for both discrete and continuous distributions),
• offer a first overview of Bayesian statistical methods, and
• demonstrate corresponding computational procedures in R.

### Learning outcomes

On the successful completion of the course, students will be able to:

• calculate marginal distributions of and conditional distributions associated with multivariate random variables
• describe a range of parametric families to model their probability distribution
• calculate expectations and conditional expectations
• evaluate the distribution of functions of random variables
• use the properties of generating functions to derive moments of distributions, distributions of sums of random variables and limit distributions for sequences of random variables.
• apply model-based approaches in statistical data analysis;
• derive maximum likelihood estimates and compute corresponding confidence intervals;
• perform statistical testing from a likelihood perspective;
• solve standard modelling and inference problems from a Bayesian point of view;
• use R to apply techniques from the course on actual data.

### Syllabus

Syllabus:
Semester 1:

Chapter 1: Random Variables (4 lectures) Definition of Events and their probabilities; definition of Random variables and their distributions; features of Discrete and Continuous random variables; Functions of random variables and mixed random variables.

Chapter 2: Multivariate random variables (6 lectures) Bivariate Distributions; Independence; Sums of several variables; Conditional distributions; the bivariate transform.

Chapter 3: Expectation (6 lectures) Expectation of a univariate random variable; Variance and higher Moments; Expectation of a bivariate random variable and conditional expectation; Probability generating functions; Moment generating functions; Sums of random variables using generating functions.

Chapter 4: Sampling and convergence (6 lectures); The sample mean; Central limit theorem; Chebyshev's Inequality; Poisson Limit Theorem and characteristic functions; Introduction to the multivariate normal distribution.

Semester 2:

Section I - Likelihood

- Entropy foundations: Shannon and differential entropy, cross-entropy, Kullback-Leibler (KL) divergence, expected Fisher information, minimum KL divergence and maximum likelihood.

- Likelihood-based estimation: Likelihood function, regular models, score function, maximum likelihood estimators (MLE), invariance principle, relationship to ordinary least-squares estimation (OLS), observed Fisher information.

- Quadratic approximation and normal asymptotics: Quadratic approximation of log-likelihood function and normal distribution, quantifying the uncertainty of MLEs using Fisher information, (squared) Wald statistic, normal confidence intervals, non-regular models.

-  Likelihood-based inference: Likelihood-based confidence interval, Wilks log-likelihood ratio statistic, likelihood ratio test, generalised likelihood ratio test, optimality properties.

Section II - Bayes

- Conditioning and Bayes rule:  Conditional probability, Bayes’ theorem, conditional mean and variance, conditional entropy and chain rules, complete data log-likelihood, observed data log-likelihood, learning unobservable states using Bayes theorem.

- Principles of Bayesian learning:  Prior and posterior probabilities and densities over parameters, marginal likelihood, sequential updates, summaries of posterior distributions and credible intervals, Bayesian and frequentist interpretation of probability.

- Standard models: Beta-binomial model (for a proportion), normal-normal model (for the mean), inverse-gamma-normal model (for the variance), properties of Bayesian learning.

- Bayesian model comparison: Log-marginal likelihood as penalised likelihood, model complexity, Bayes factor, Schwarz approximation and Bayesian Information Criterion (BIC), Bayesian testing using false discovery rate.

- Choosing priors and optimality properties: default priors, uninformative priors, empirical Bayes, shrinkage estimation, James-Stein estimator, Frequentist properties of Bayesian estimators, optimality of Bayes inference (e.g. Cox theorem).

### Teaching and learning methods

This course unit forms the core of Theme C, and as such is delivered over 2 semesters.  Teaching is composed of two hours of lectures per week, and one tutorial class per fortnight.  Teaching materials will be uploaded to Blackboard for reference and review.

### Assessment methods

Method Weight
Other 20%
Written exam 80%

Other: one mid-term online timed test and one mid-term written test, 40-50 mins each.

### Feedback methods

Marked scripts within a week of the in-class test and automatically given feedback following the online test. Generic feedback made available after marks are released.

Semester 1:

Mood, A. M., Graybill, F. A. and Boes, D. C., Introduction to the Theory of Statistics, 3rd edition, McGraw-Hill 1974

S. Ross, A First Course in Probability, 4th edition, Macmillan.

D. Stirzaker, Elementary Probability, Cambridge University Press. Available electronically

Neil A. Weiss, A Course in Probability, Pearson.

Semester 2:

Strimmer, K. 2024. MATH27720 Part 2 lecture notes. (Essential)

Held, L, and Bove, D.S. 2020.  Applied Statistical Inference: Likelihood and Bayes (2nd edition). Springer (Recommended)

### Study hours

Scheduled activity hours
Lectures 44
Tutorials 12
Independent study hours
Independent study 144

### Teaching staff

Staff member Role
Robert Gaunt Unit coordinator
Korbinian Strimmer Unit coordinator
Yuk Ka Chung Unit coordinator