Markov Processes

The goal of the course is to get the students aquainted with the stochastic processes with `no' memory (Markov processes), where the probability of any future event only depends on the current state of the process. Such processes are Markov processes or Markov chains. Markov chains have many applications as statistical models of real-world processes. They provide the basis for general stochastic simulation methods known as Markov chain Monte Carlo, which are used for simulating sampling from complex probability distributions, and have found application in areas including Bayesian statistics, biology, chemistry, economics, finance, information theory, physics, signal processing, and speech processing.

Topics that will be covered include: discrete and continuous time Markov chains, transition matrix, classification of states, irreducibility and periodicity, invariant measure, return times and hitting times, long-term behavior of the system. Special Markov chains on countable state spaces: Branching processes, (survival vs extinction) Birth and death chains (recurrence, transience), Poisson point process. Random Walk on the integers Z and other graphs.

Moreover, the course will introduce Markov decision processes, which are a mathematical support tool for decision making under uncertainty. The mathematical concepts will be presented, and application of Markov decision processes to concrete decision contexts will be considered. Two real-life motivated decision making problems (in inventory management and replacement) will be modelled and solved with Markov decision processes.