Classification of States

Abstract

In this chapter we present the notions of communicating, transient and recurrent states, as well as the concept of irreducibility of a Markov chain. We also examine the notions of positive and null recurrence, periodicity, and aperiodicity of such chains. Those topics will be important when analysing the long-run behavior of Markov chains in the next chapter.

Exercises

Exercise 6.1

Consider a Markov chain $(X_n)_{n\ge 0}$ on the state space $\{ 0,1,2,3 \}$, with transition matrix

$$ \left[ \begin{array}{cccc} 1/3 ~&{} 1/3 ~&{} 1/3 ~&{} 0 \\ 0 ~&{} 0 ~&{} 0 ~&{} 1 \\ 0 ~&{} 1 ~&{} 0 ~&{} 0 \\ 0 ~&{} 0 ~&{} 1 ~&{} 0 \end{array} \right] . $$

• (a) Draw the graph of this chain and find its communicating classes. Is this Markov chain reducible? Why?

• (b) Find the periods of states , , , and .

• (c) Compute $\mathbb {P}(T_0<\infty \mid X_0=0)$, $\mathbb {P}(T_0 = \infty \mid X_0=0)$, and $\mathbb {P}(R_0<\infty \mid X_0=0)$.

• (d) Which state(s) is (are) absorbing, recurrent, and transient?

Exercise 6.2

Consider the Markov chain on $\{ 0 , 1 , 2 \}$ with transition matrix

$$ \left[ \begin{array}{ccc} 1/3 ~&{} 1/3 ~&{} 1/3 \\ 1/4 ~&{} 3/4 ~&{} 0 \\ 0 ~&{} 0 ~&{} 1 \end{array} \right] . $$

• (a) Is the chain irreducible? Give its communicating classes.

• (b) Which states are absorbing, transient, recurrent, positive recurrent?

• (c) Find the period of every state.

Exercise 6.3

Consider a Markov chain $(X_n)_{n\ge 0}$ on the state space $\{ 0,1,2,3,4\}$, with transition matrix

$$ \left[ \begin{array}{ccccc} 0 &{} ~1/4 &{} ~1/4 &{} ~1/4 &{} ~1/4 \\ 1 &{} 0 &{} 0 &{} 0 &{} 0 \\ 0 &{} 1 &{} 0 &{} 0 &{} 0 \\ 0 &{} 0 &{} 1 &{} 0 &{} 0 \\ 0 &{} 0 &{} 0 &{} 0 &{} 1 \end{array} \right] . $$

• (a) Draw the graph of this chain.

• (b) Find the periods of states , , , and .

• (c) Which state(s) is (are) absorbing, recurrent, and transient?

• (d) Is the Markov chain reducible? Why?

Exercise 6.4

Consider the Markov chain with transition matrix

$$ \left[ ~ P_{i, j} ~ \right] _{ 0 \le i , j \le 5 } = \left[ \begin{array}{cccccc} 1/2 &{} ~0 &{} ~1/4 &{} ~0 &{} ~0 &{} ~1/4 \\ 1/3 &{} ~1/3 &{} ~1/3 &{} ~0 &{} ~0 &{} ~0 \\ 0 &{} ~0 &{} ~0 &{} ~0 &{} ~1 &{} ~0 \\ 1/6 &{} ~1/2 &{} ~1/6 &{} ~0 &{} ~0 &{} ~1/6 \\ 0 &{} ~0 &{} ~1 &{} ~0 &{} ~0 &{} ~0 \\ 0 &{} ~0 &{} ~0 &{} ~0 &{} ~0 &{} ~1 \end{array} \right] . $$

• (a) Is the chain reducible? If yes, find its communicating classes.

• (b) Determine the transient and recurrent states of the chain.

• (c) Find the period of each state.

Exercise 6.5

Consider the Markov chain with transition matrix

$$ \left[ \begin{array}{cccc} 0.8 ~&{} 0 ~&{} 0.2 ~&{} 0 \\ 0 ~&{} 0 ~&{} 1 ~&{} 0 \\ 1 ~&{} 0 ~&{} 0 ~&{} 0 \\ 0.3 ~&{} 0.4 ~&{} 0 ~&{} 0.3 \end{array} \right] . $$

• (a) Is the chain irreducible? If not, give its communicating classes.

• (b) Find the period of each state. Which states are absorbing, transient, recurrent, positive recurrent?

Exercise 6.6

In the following chain, find:

• (a) the communicating class(es),

• (b) the transient state(s),

• (c) the recurrent state(s),

• (d) the positive recurrent state(s),

• (e) the period of every state.

Exercise 6.7

Consider two boxes containing a total of N balls. At each unit of time one ball is chosen randomly among N and moved to the other box.

• (a) Write down the transition matrix of the Markov chain $(X_n)_{n\in {\mathord {\mathbb N}}}$ with state space $\{ 0, 1, 2, \ldots , N\}$, representing the number of balls in the first box.

• (b) Determine the periodicity, transience and recurrence of the Markov chain.

Exercise 6.8

• (a) Is the Markov chain of Exercise 4.10-(a) recurrent? positive recurrent?

• (b) Find the periodicity of every state.

• (c) Same questions for the success runs Markov chain of Exercise 4.10-(b).

Problem 6.9

Let $\alpha > 0$ and consider the Markov chain with state space ${\mathord {\mathbb N}}$ and transition matrix given by

$$ P_{i, i-1} = \frac{1}{\alpha + 1}, \quad P_{i, i+1} = \frac{\alpha }{\alpha + 1}, \qquad i \ge 1. $$

and a reflecting barrier at 0, such that $P_{0,1}=1$. Compute the mean return times $\mathrm{I}\! \mathrm{E}[ T^r_k \mid X_0 = k ]$ for $k\in {\mathord {\mathbb N}}$, and show that the chain is positive recurrent if and only if $\alpha < 1$.