Appendix
First, we provide an important technical result that is exploited in some of the proofs below.
Lemma 5
(Puterman 1994) Let \(\{x_j\}\), \(\{x_j'\}\) be real-valued non-negative sequences satisfying
$$\begin{aligned} \sum _{j=k}^{\infty } x_j \ge \sum _{j=k}^{\infty } x_j' \end{aligned}$$
(15)
for all k, with equality holding in (15) for \(k = 0\). Suppose \(v_{j+1} \ge v_j\) for \(j = 0,1,\dots ,\) then
$$\begin{aligned} \sum _{j=0}^{\infty } v_j x_j \ge \sum _{j=0}^{\infty } v_j x_j', \end{aligned}$$
(16)
where limits in (16) exist but may be infinite.
Proof of Lemma 1
Assume \(\delta _1 \ge \delta _2\). By Assumption A4, we have
$$\begin{aligned}&\sum _{\delta '=k}^{D} P^{(d)}(\delta '|\delta _1) \ge \sum _{\delta '=k}^{D} P^{(d)}(\delta '|\delta _2) \text{, } \text{ and } \\&{\sum _{\delta '=k}^{D} P_\theta ^{(d)}(\delta '|\delta _1) \ge \sum _{\delta '=k}^{D} P_\theta ^{(d)}(\delta '|\delta _2)} \end{aligned}$$
for all \(k \in \Delta \), \(\theta \in \Theta \) and \(d \in \mathbb {Z}_+\). Recall that by Assumption A3, \(q^\theta (\delta )\) is nonincreasing in \(\delta \). Then, by Lemma 5, it follows that
$$\begin{aligned}&\sum _{\delta '=1}^{D} P^{(j)}(\delta '|\delta _1) (-q^{\theta }(\delta ')) \ge \sum _{\delta '=1}^{D} P^{(j)}(\delta '|\delta _2) (-q^{\theta }(\delta ')),\quad \forall j \in \{1,2,\dots ,d^{\theta }\} \text{, } \text{ and } \\&{\sum _{\delta '=1}^{D} P_\theta ^{(j)}(\delta '|\delta _1) (-q^{\theta }(\delta ')) \ge \sum _{\delta '=1}^{D} P_\theta ^{(j)}(\delta '|\delta _2) (-q^{\theta }(\delta ')),\quad \forall j \in \{1,2,\dots ,d^{\theta }\}.} \end{aligned}$$
Therefore, \(r_0(\theta ,\delta _1) \le r_0(\theta ,\delta _2)\) and \(r_1(\theta ,\delta _1) \le r_1(\theta ,\delta _2)\). \(\square \)
Proof of Theorem 1
Next, we establish conditions under which it is optimal to reserve the palliative intervention as a last resort. Let us first construct a general sequence as follows: there is a delay x units of time before the system starts intervention A (i.e., the system is on palliative intervention for x units of time). If A fails, then there is another delay y units of time before the system switches to intervention B. Here, we omit the part of a sequence after both A and B fail, because the system is put on the palliative intervention indefinitely then. Note that x and y are both nonnegative integers. Denote the corresponding sequence as \(S_1\). We also construct two additional sequences \(S_2\) and \(S_3\) in a similar manner, see Fig. 5. Our goal is to show that the rewards generated by those three sequences are ordered as \(R^{\langle S_1 \rangle } \le R^{\langle S_2 \rangle } \le R^{\langle S_3 \rangle }\).
First, observe that if intervention A is ineffective, then when A reveals this fact (i.e., when the system reaches age \(t_0 + d^A + x\)), the distribution of the deterioration level is the same for both sequences \(S_1\) and \(S_2\), since the system transitions according to P during this time. As a result, the expected reward generated from this point forward is the same for both sequences. To order \(R^{\langle S_1 \rangle }\) and \(R^{\langle S_2 \rangle }\), we need to consider two cases: (i) the reward generated if intervention A is effective; (ii) the reward generated between \((t_0,t_0 + d^A + x)\) if A is ineffective. The expected reward for cases (i) and (ii) in sequence \(S_1\) is given by:
$$\begin{aligned}&\sum _{j=1}^{x} \sum _{\delta '=1}^{D} P^{(j)}(\delta '|\delta _{t_0}) q^{\theta ^M}(\delta ',t_0+j) + \rho ^A\cdot \sum _{\delta '=1}^{D} P^{(x)}(\delta '|\delta _{t_0}) r_1(A,\delta ') \nonumber \\&\quad +\, \rho ^A Q^A_E\left( t_0+d^A+x\right) +\,{(1-\rho ^A)\cdot \sum _{\delta '=1}^{D} P^{(x)}(\delta '|\delta _{t_0}) r_0(A,\delta ').} \end{aligned}$$
(17)
Similarly, the expected reward for cases (i) and (ii) in sequence \(S_2\) is:
$$\begin{aligned}&{\rho ^A\cdot r_1(A,\delta _{t_0}) + (1-\rho ^A)\cdot r_0(A,\delta _{t_0})} \nonumber \\&\quad {+\, (1-\rho ^A)\sum _{j=d^A+1}^{d^A+x} \sum _{\delta '=1}^{D} P^{(j)}(\delta '|\delta _{t_0}) q^{\theta ^M}(\delta ',t_0+j) + \rho ^A Q^A_E\left( t_0+d^A\right) .} \end{aligned}$$
(18)
Subtracting (17) from (18), we have
$$\begin{aligned}&{\rho ^A \bigg (Q^A_E\left( t_0+d^A\right) -Q^A_E\left( t_0+d^A+x\right) \bigg ) - \sum _{j=1}^{x} \sum _{\delta '=1}^{D} P^{(j)}(\delta '|\delta _{t_0}) q^{\theta ^M}(\delta ',t_0+j) + (1-\rho ^A)} \nonumber \\&\quad \times {\sum _{j=d^A+1}^{d^A+x} \sum _{\delta '=1}^{D} P^{(j)}(\delta '|\delta _{t_0}) q^{\theta ^M}(\delta ',t_0+j) + \rho ^A\cdot \bigg (r_1(A,\delta _{t_0}) - \sum _{\delta '=1}^{D} P^{(x)}(\delta '|\delta _{t_0}) r_1(A,\delta ')\bigg ) } \nonumber \\&\quad + {(1-\rho ^A)\cdot \bigg (r_0(A,\delta _{t_0}) - \sum _{\delta '=1}^{D} P^{(x)}(\delta '|\delta _{t_0}) r_0(A,\delta ')\bigg )} \end{aligned}$$
(19)
that represents the obtained difference. Based on Assumption A4, Lemma 1 and Lemma 5, observe that
$$\begin{aligned}&{r_1(A,\delta _{t_0}) - \sum _{\delta '=1}^{D} P^{(x)}(\delta '|\delta _{t_0}) r_1(A,\delta ') \ge 0,\ \text{ and }} \nonumber \\&{r_0(A,\delta _{t_0}) - \sum _{\delta '=1}^{D} P^{(x)}(\delta '|\delta _{t_0}) r_0(A,\delta ') \ge 0.} \end{aligned}$$
(20)
As a result, if
$$\begin{aligned} \rho ^A \bigg (Q^A_E\left( t_0+d^A\right) -Q^A_E\left( t_0+d^A+x\right) \bigg ) \ge \sum _{j=1}^{x} \sum _{\delta '=1}^{D} P^{(j)}(\delta '|\delta _{t_0}) q^{\theta ^M}(\delta ',t_0+j), \end{aligned}$$
(21)
then (19) is nonnegative, i.e., the reward generated in sequence \(S_2\) is no less than that in \(S_1\). We conclude that under condition (21), sequence \(S_1\) results in a lower expected reward than \(S_2\).
Now, let us prove that the reward generated by \(S_2\) is lower than that by \(S_3\). Observe that both sequences start with intervention A for \(d^A\) units of time. Therefore, we simply omit this period from consideration. The distribution of the deterioration level of the system is the same in both new sequences \(S_2\) and \(S_3\). We immediately identify a similar situation as in the previous step (i.e., the system is in the absence of one-time intervention for \(x+y\) units of time before starting intervention B). By applying the same argument as in (17)–(21), we conclude that if
$$\begin{aligned} \rho ^B \bigg (Q^B_E\left( t_0+d^B\right) -Q^B_E\left( t_0+d^B+x+y\right) \bigg ) \ge \sum _{j=1}^{x+y} \sum _{\delta '=1}^{D} P^{(j)}(\delta '|\delta _{t_0}) q^{\theta ^M}(\delta ',t_0+j), \end{aligned}$$
(22)
then sequence \(S_2\) leads to a lower expected reward than \(S_3\). \(\square \)
Proof of Theorem 2
Without loss of generality, assume that \(\langle AB \rangle \) is an optimal sequence. By Lemma 1, \(f_1^{\langle AB \rangle }(\delta )\) is nonincreasing in \(\delta \). By applying Lemma 5 and mimicking the proof of Lemma 1, we can also prove that \(f_2^{\langle AB \rangle }(\delta )\) is nonincreasing in \(\delta \). Similarly, we can show that \(f_3^{\langle AB \rangle }(\delta )\) is nonincreasing in \(\delta \) by exploiting Assumption A5. Since \(V(\delta )\) is a linear combination of \(f_1^{\langle AB \rangle }(\delta )\), \(f_2^{\langle AB \rangle }(\delta )\) and \(f_3^{\langle AB \rangle }(\delta )\) with nonnegative coefficients, the nonincreasing property of \(V(\delta )\) in \(\delta \) follows. \(\square \)
Proof of Lemma 2
First, we prove that
$$\begin{aligned} f_1^{\langle AB \rangle }(\delta _{t_0}) \ge f_1^{\langle BA \rangle }(\delta _{t_0}). \end{aligned}$$
(23)
Based on Assumption A3 and the fact that \(q^A(\delta ) \ge q^B(\delta )\) for all \(\delta \in \Delta \), and \(Q^A_E(t) \ge Q^B_E(t)\) for all t, we observe that
$$\begin{aligned} Q^A_E\left( t_0+d^A\right) - Q^A_E\left( t_0+d^B\right) \ge \max _\delta q^A(\delta )\cdot (d^B-d^A) \ge \max _\delta q^B(\delta )\cdot (d^B-d^A). \end{aligned}$$
(24)
As \(d^B \ge d^A\) and \(q^A(\delta ) \ge q^B(\delta )\) for all \(\delta \in \Delta \), from (2) we have
$$\begin{aligned} r(B,\delta _{t_0}) =&\sum _{j=1}^{d^B} \sum _{\delta '=1}^{D} P^{(j)}(\delta '|\delta _{t_0}) q^{B}(\delta ') \nonumber \\ =&\sum _{j=1}^{d^A} \sum _{\delta '=1}^{D} P^{(j)}(\delta '|\delta _{t_0}) q^{B}(\delta ') + \sum _{j=d^A+1}^{d^B} \sum _{\delta '=1}^{D} P^{(j)}(\delta '|\delta _{t_0}) q^{B}(\delta ') \nonumber \\ \le&\sum _{j=1}^{d^A} \sum _{\delta '=1}^{D} P^{(j)}(\delta '|\delta _{t_0}) q^{A}(\delta ') + \max _\delta q^B(\delta ) \cdot \sum _{j=d^A+1}^{d^B} \sum _{\delta '=1}^{D} P^{(j)}(\delta '|\delta _{t_0}) \nonumber \\ =&\sum _{j=1}^{d^A} \sum _{\delta '=1}^{D} P^{(j)}(\delta '|\delta _{t_0}) q^{A}(\delta ') + \max _\delta q^B(\delta )\cdot (d^B-d^A) \end{aligned}$$
(25)
$$\begin{aligned} \le&r(A,\delta _{t_0}) + Q^A_E\left( t_0+d^A\right) - Q^A_E\left( t_0+d^B\right) , \end{aligned}$$
(26)
where the last inequality follows from (24). As \(Q^A_E\left( t_0+d^B\right) \ge Q^B_E\left( t_0+d^B\right) \), then (26) implies that
$$\begin{aligned} r(A,\delta _0) + Q^A_E\left( t_0+d^A\right)&= r(A,\delta _0) + Q^A_E\left( t_0+d^A\right) - Q^A_E\left( t_0+d^B\right) + Q^A_E\left( t_0+d^B\right) \\&\quad \,\ge r(B,\delta _{t_0}) + Q^B_E\left( t_0+d^B\right) , \end{aligned}$$
and therefore, (23) holds.
Next, we show that
$$\begin{aligned} f_1^{\langle BA \rangle }(\delta _{t_0}) \ge f_2^{\langle AB \rangle }(\delta _{t_0}). \end{aligned}$$
From Lemma 1 and Assumption A4, we observe that
$$\begin{aligned} r(B,\delta _{t_0}) \ge \sum _{\delta =1}^D P^{(d^A)}(\delta |\delta _{t_0})\cdot r(B,\delta ) . \end{aligned}$$
(27)
Furthermore, we have
$$\begin{aligned}&Q^B_E\left( t_0+d^B\right) - Q^B_E\left( t_0+d^A+d^B\right) - r(A,\delta _{t_0}) \nonumber \\&\qquad \qquad \ge \max _\delta q^A(\delta ) \cdot d^A - r(A,\delta _{t_0}) \ge 0, \end{aligned}$$
(28)
where the second inequality is by Assumption A3. Therefore, by applying (27) and (28) we obtain that
$$\begin{aligned}&f_1^{\langle BA \rangle }(\delta _{t_0}) - f_2^{\langle AB \rangle }(\delta _{t_0})\\&\quad = \bigg (r(B,\delta _{t_0}) - \sum _{\delta =1}^D P^{(d^A)}(\delta |\delta _{t_0})\cdot r(B,\delta )\bigg )\\&\qquad +\, \bigg (Q^B_E\left( t_0+d^B\right) - Q^B_E\left( t_0+d^A+d^B\right) - r(A,\delta _{t_0}) \bigg ) \ge 0, \end{aligned}$$
which completes the proof. \(\square \)
Proof of Lemma 3
Define
$$\begin{aligned} L(u)=\sum _{j=1}^{d^A} \sum _{\delta '=1}^{D} P^{(j)}(\delta '|\delta _{t_0}) q^{A}(\delta ') + \sum _{\delta '=1}^{D} P^{(d^A)}(\delta '|\delta _{t_0}) \sum _{j=1}^{u} \sum _{\delta ''=1}^{D} P^{(j)}(\delta ''|\delta ') q^{B}(\delta ''). \end{aligned}$$
(29)
Similarly, define
$$\begin{aligned} R(u)=\sum _{j=1}^{u} \sum _{\delta '=1}^{D} P^{(j)}(\delta '|\delta _{t_0}) q^{B}(\delta ') + \sum _{\delta '=1}^{D} P^{(u)}(\delta '|\delta _{t_0}) \sum _{j=1}^{d^A} \sum _{\delta ''=1}^{D} P^{(j)}(\delta ''|\delta ') q^{A}(\delta ''). \end{aligned}$$
(30)
Observe that (9) is equivalent to
$$\begin{aligned} L(d^B) - R(d^B) \ge 0. \end{aligned}$$
(31)
Next, we prove (31). For any \(k,j \in \mathbb {Z}_+\) and \(\delta '' \in \Delta \), we have (e.g., see Serfozo (2009))
$$\begin{aligned} \sum _{\delta '=1}^{D} P^{(k)}(\delta '|\delta _{t_0}) P^{(j)}(\delta ''|\delta ') = P^{(k+j)}(\delta ''|\delta _{t_0}), \end{aligned}$$
which implies that
$$\begin{aligned} \sum _{\delta '=1}^{D} P^{(d^A)}(\delta '|\delta _{t_0}) \sum _{j=1}^{u} \sum _{\delta ''=1}^{D} P^{(j)}(\delta ''|\delta ') q^{B}(\delta '') = \sum _{j=d^A+1}^{d^A+u} \sum _{\delta =1}^{D} P^{(j)}(\delta |\delta _{t_0}) q^{B}(\delta ), \end{aligned}$$
and
$$\begin{aligned} \sum _{\delta '=1}^{D} P^{(u)}(\delta '|\delta _{t_0}) \sum _{j=1}^{d^A} \sum _{\delta ''=1}^{D} P^{(j)}(\delta ''|\delta ') q^{A}(\delta '') = \sum _{j=u+1}^{d^A+u} \sum _{\delta =1}^{D} P^{(j)}(\delta |\delta _{t_0}) q^{A}(\delta ). \end{aligned}$$
Therefore,
$$\begin{aligned} L(d^A) - R(d^A)&= \sum _{j=1}^{d^A} \sum _{\delta =1}^{D} P^{(j)}(\delta |\delta _{t_0}) (q^{A}(\delta ) - q^{B}(\delta ))\nonumber \\&\quad \,-\sum _{j=d^A+1}^{d^A+d^A} \sum _{\delta =1}^{D} P^{(j)}(\delta |\delta _{t_0}) (q^{A}(\delta ) - q^{B}(\delta )) \ge 0, \end{aligned}$$
(32)
where the inequality holds by Assumption A4. Because both \(d^A\) and \(d^B\) are integers, and \(d^B \ge d^A\), we can now increase u from \(d_A\) to \(d_B\) and check if a similar inequality still holds as in (32). If \(u = d^A + 1\), then the increase in (29) is given by
$$\begin{aligned} L(d^A+1) - L(d^A)&= \sum _{\delta =1}^{D} \sum _{j=d^A+1}^{d^A+d^A+1} P^{(j)}(\delta |\delta _{t_0}) q^{B}(\delta ) \\&\quad - \sum _{\delta =1}^{D} \sum _{j=d^A+1}^{d^A+d^A} P^{(j)}(\delta |\delta _{t_0}) q^{B}(\delta ) \nonumber \\&= \sum _{\delta =1}^{D} P^{(d^A+d^A+1)}(\delta |\delta _{t_0}) q^{B}(\delta ), \end{aligned}$$
and the increase in (30) is given by
$$\begin{aligned}&R(d^A+1) - R(d^A) \\&\quad = \sum _{\delta =1}^{D} P^{(d^A+1)}(\delta |\delta _{t_0}) q^{B}(\delta ) + \sum _{j=d^A+2}^{d^A+d^A+1} \sum _{\delta =1}^{D} P^{(j)}(\delta |\delta _{t_0}) q^{A}(\delta ) \\&\qquad -\, \sum _{j=d^A+1}^{d^A+d^A} \sum _{\delta =1}^{D} P^{(j)}(\delta |\delta _{t_0}) q^{A}(\delta ) \\&\quad = \sum _{\delta =1}^{D} P^{(d^A+1)}(\delta |\delta _{t_0}) q^{B}(\delta ) + \sum _{\delta =1}^{D} P^{(d^A+d^A+1)}(\delta |\delta _{t_0}) q^{A}(\delta ) - \sum _{\delta =1}^{D} P^{(d^A+1)}(\delta |\delta _{t_0}) q^{A}(\delta ). \end{aligned}$$
Then by Lemma 5 we have that
$$\begin{aligned}&L(d^A+1) - L(d^A) - \bigg (R(d^A+1) - R(d^A)\bigg ) \nonumber \\&\quad = \sum _{\delta =1}^{D} P^{(d^A+d^A+1)}(\delta |\delta _{t_0}) (q^B(\delta )-q^A(\delta )) - \sum _{\delta =1}^{D} P^{(d^A+1)}(\delta |\delta _{t_0}) (q^B(\delta )-q^A(\delta )) \ge 0. \end{aligned}$$
(33)
Therefore,
$$\begin{aligned}&L(d^A+1)- R(d^A+1) \\&\quad = \bigg (L(d^A+1) - L(d^A)+ L(d^A)\bigg ) - \bigg (R(d^A+1) - R(d^A)+R(d^A)\bigg ) \\&\quad = \bigg (L(d^A+1) - L(d^A) - R(d^A+1) + R(d^A) \bigg ) + \bigg (L(d^A)-R(d^A)\bigg ) \ge 0. \end{aligned}$$
The last inequality holds due to (32) and (33). It implies that increasing one unit of u in both (29) and (30) does not change the inequality in (32). We can apply the same approach to u until it reaches \(d^B\), and the same result holds. Thus, (31) holds, which completes the proof. \(\square \)
Proof of Lemma 4
By taking the difference between the left-hand and right-hand sides of (10), we have
$$\begin{aligned}&r(A,\delta _{t_0}) + Q^A_E\left( t_0 + d^A\right) - \bigg (r(B,\delta _{t_0}) + \sum _{\delta =1}^{D}P^{(d^B)} (\delta |\delta _{t_0}) \cdot r(A,\delta ) + Q^A_E\left( t_0 + d^A + d^B\right) \bigg ) \nonumber \\&\quad = \bigg (r(A,\delta _{t_0}) - \sum _{\delta =1}^{D}P^{(d^B)} (\delta |\delta _{t_0}) \cdot r(A,\delta )\bigg ) \nonumber \\&\qquad +\, \bigg (Q^A_E\left( t_0+d^A\right) - Q^A_E\left( t_0+d^A+d^B\right) - r(B,\delta _{t_0}) \bigg ) . \end{aligned}$$
(34)
Observe that \(r(A,\delta _{t_0}) \ge \sum _{\delta =1}^{D}P^{(d^B)} (\delta |\delta _{t_0}) \cdot r(A,\delta )\), because P is upper triangular by Assumption A4. In addition, by Assumption A3
$$\begin{aligned} Q^A_E\left( t_0+d^A\right) - Q^A_E\left( t_0+d^A+d^B\right) \ge \max _{\delta } q^A(\delta ) d^B \ge \max _{\delta } q^B(\delta ) d^B \ge r(B,\delta _{t_0}). \end{aligned}$$
Therefore, (34) is nonnegative, which completes the proof. \(\square \)
Proof of Theorem 3
First, from Assumption A3 and similar to (24) we observe that
$$\begin{aligned}&r(A,\delta _{t_0}) + \bigg (Q^B_E\left( t_0 + d^A\right) - Q^B_E\left( t_0+d^B\right) \bigg ) \nonumber \\&\ge r(A,\delta _{t_0}) + \max _\delta q^B(\delta )\cdot (d^B-d^A) \nonumber \\&\ge \sum _{j=1}^{d^A} \sum _{\delta '=1}^{D} P^{(j)}(\delta '|\delta ) q^{B}(\delta ') + \max _\delta q^B(\delta )\cdot (d^B-d^A) \nonumber \\&\ge r(B,\delta _{t_0}) . \end{aligned}$$
(35)
Next, we have
$$\begin{aligned}&r(A,\delta _{t_0}) + Q^A_E\left( t_0 + d^A\right) - Q^A_E\left( t_0+d^A+d^B\right) \nonumber \\&\quad = r(A,\delta _{t_0}) + \bigg (Q^A_E\left( t_0 + d^A\right) - Q^A_E\left( t_0+d^A+d^B\right) \bigg )\nonumber \\&\qquad + \bigg (Q^B_E\left( t_0 + d^B\right) - Q^B_E\left( t_0+d^A+d^B\right) \bigg ) \nonumber \\&\qquad -\, \bigg (Q^B_E\left( t_0 + d^B\right) - Q^B_E\left( t_0+d^A+d^B\right) \bigg ) \nonumber \\&\ge r(A,\delta _{t_0}) + \bigg (Q^B_E\left( t_0 + d^A\right) - Q^B_E\left( t_0+d^A+d^B\right) \bigg ) \nonumber \\&\qquad + \bigg (Q^B_E\left( t_0 + d^B\right) - Q^B_E\left( t_0+d^A+d^B\right) \bigg ) \nonumber \\&\qquad - \,\bigg (Q^B_E\left( t_0 + d^B\right) - Q^B_E\left( t_0+d^A+d^B\right) \bigg ) \nonumber \\&= r(A,\delta _{t_0}) + \bigg (Q^B_E\left( t_0 + d^A\right) - Q^B_E\left( t_0+d^B\right) \bigg )\nonumber \\&\qquad + \bigg (Q^B_E\left( t_0 + d^B\right) - Q^B_E\left( t_0+d^A+d^B\right) \bigg ) \nonumber \\&\ge r(B,\delta _{t_0}) + Q^B_E\left( t_0 + d^B\right) - Q^B_E\left( t_0+d^A+d^B\right) , \end{aligned}$$
(36)
where the first inequality holds by the assumption that \(Q^A_E(t) - Q^A_E(t+k) \ge Q^B_E(t) - Q^B_E(t+k)\) for all t, k, and the second inequality holds by (35). Therefore, (36) implies that
$$\begin{aligned}&r(A,\delta _{t_0}) + Q^A_E\big (t_0 + d^A\big ) + Q^B_E\big (t_0+d^A+d^B\big ) \nonumber \\&\quad \ge r(B,\delta _{t_0}) + Q^B_E\big (t_0 + d^B\big ) + Q^A_E\big (t_0+d^A+d^B\big ). \end{aligned}$$
(37)
Next, observe that
$$\begin{aligned}&f_1^{\langle AB \rangle }(\delta _{t_0}) + f_2^{\langle AB \rangle }(\delta _{t_0}) - \bigg ( f_1^{\langle BA \rangle }(\delta _{t_0}) + f_2^{\langle BA \rangle }(\delta _{t_0}) \bigg ) \nonumber \\&= r(A,\delta _{t_0}) + Q^A_E\big (t_0+d^A\big ) + r(A,\delta _{t_0}) + \sum _{\delta =1}^{D}P^{(d^A)} (\delta |\delta _{t_0}) \cdot r(B,\delta ) + Q^B_E\big (t_0+d^A+d^B\big ) \nonumber \\&\quad -\, \bigg (r(B,\delta _{t_0}) + Q^B_E\big (t_0+d^B\big ) + r(B,\delta _{t_0}) + \sum _{\delta =1}^{D}P^{(d^B)} (\delta |\delta _{t_0}) \cdot r(A,\delta ) + Q^A_E\big (t_0+d^A+d^B\big ) \bigg ) \nonumber \\&= \bigg (r(A,\delta _{t_0}) + Q^A_E\big (t_0+d^A\big )+Q^B_E\big (t_0+d^A+d^B\big ) - r(B,\delta _{t_0}) \nonumber \\&\qquad - Q^B_E\big (t_0+d^B\big ) - Q^A_E\big (t_0+d^A+d^B\big ) \bigg ) \nonumber \\&\quad +\, \bigg (r(A,\delta _{t_0}) + \sum _{\delta =1}^{D}P^{(d^A)} (\delta |\delta _{t_0}) \cdot r(B,\delta ) - r(B,\delta _{t_0}) - \sum _{\delta =1}^{D}P^{(d^B)} (\delta |\delta _{t_0}) \cdot r(A,\delta ) \bigg ) \ge 0, \end{aligned}$$
(38)
where the last inequality holds by (37) and Lemma 3. Also, from Lemma 3, we have that
$$\begin{aligned}&f_3^{\langle AB \rangle }(\delta _{t_0})\\&\quad = r(A,\delta _{t_0}) + \sum _{\delta =1}^{D}P^{(d^A)} (\delta |\delta _{t_0}) \cdot r(B,\delta ) + \sum _{\delta =1}^{D}P^{(d^A+d^B)} (\delta |\delta _{t_0}) Q^{\theta ^M}\big (\delta ,t_0+d^A+d^B\big ) \\&\quad \ge r(B,\delta _{t_0}) + \sum _{\delta =1}^{D}P^{(d^B)} (\delta |\delta _{t_0}) \cdot r(A,\delta ) + \sum _{\delta =1}^{D}P^{(d^A+d^B)} (\delta |\delta _{t_0}) Q^{\theta ^M}\big (\delta ,t_0+d^A+d^B\big ) \\&\quad = f_3^{\langle BA \rangle }(\delta _{t_0}). \end{aligned}$$
Therefore, from (7), we observe that a sufficient condition under which it is optimal to prescribe intervention A first is given by
$$\begin{aligned} \rho ^A f_1^{\langle AB \rangle }(\delta _{t_0}) + (1-\rho ^A)\rho ^B f_2^{\langle AB \rangle }(\delta _{t_0}) \ge \rho ^B f_1^{\langle BA \rangle }(\delta _{t_0}) + (1-\rho ^B)\rho ^A f_2^{\langle BA \rangle }(\delta _{t_0}), \end{aligned}$$
which can be re-written as
$$\begin{aligned}&\rho ^A \ge {\bar{\rho }} = \frac{\rho ^B \cdot \bigg (f_1^{\langle BA \rangle }(\delta _{t_0})-f_2^{\langle AB \rangle }(\delta _{t_0})\bigg )}{f_1^{\langle AB \rangle }(\delta _{t_0})-\rho ^B f_2^{\langle AB \rangle }(\delta _{t_0}) - (1-\rho ^B) f_2^{\langle BA \rangle }(\delta _{t_0})}. \end{aligned}$$
(39)
Now, let us verify that \({\bar{\rho }} \le \rho ^B\). The difference between \(f_1^{\langle BA \rangle }(\delta _{t_0})-f_2^{\langle AB \rangle }(\delta _{t_0})\) and \(f_1^{\langle AB \rangle }(\delta _{t_0})-\rho ^B f_2^{\langle AB \rangle }(\delta _{t_0}) - (1-\rho ^B) f_2^{\langle BA \rangle }(\delta _{t_0})\) in (39) is given by
$$\begin{aligned} f_1^{\langle BA \rangle }(\delta _{t_0}) + (1-\rho ^B)f_2^{\langle BA \rangle }(\delta _{t_0}) - \bigg (f_1^{\langle AB \rangle }(\delta _{t_0}) + (1-\rho ^B)f_2^{\langle AB \rangle }(\delta _{t_0})\bigg ). \end{aligned}$$
(40)
From (38) we have
$$\begin{aligned} f_1^{\langle BA \rangle }(\delta _{t_0}) + f_2^{\langle BA \rangle }(\delta _{t_0}) \le f_1^{\langle AB \rangle }(\delta _{t_0}) + f_2^{\langle AB \rangle }(\delta _{t_0}), \end{aligned}$$
and by multiplying the both sides in the above inequality by \((1-\rho ^B)\), we have
$$\begin{aligned} (1-\rho ^B) \bigg (f_1^{\langle BA \rangle }(\delta _{t_0}) + f_2^{\langle BA \rangle }(\delta _{t_0})\bigg ) \le (1-\rho ^B) \bigg (f_1^{\langle AB \rangle }(\delta _{t_0}) + f_2^{\langle AB \rangle }(\delta _{t_0})\bigg ). \end{aligned}$$
(41)
From Lemma 2, we have
$$\begin{aligned} f_1^{\langle BA \rangle }(\delta _{t_0}) \le f_1^{\langle AB \rangle }(\delta _{t_0}). \end{aligned}$$
(42)
Multiplying (42) by \(\rho ^B\) and adding to (41), we obtain
$$\begin{aligned} f_1^{\langle BA \rangle }(\delta _{t_0}) + (1-\rho ^B)f_2^{\langle BA \rangle }(\delta _{t_0}) \le f_1^{\langle AB \rangle }(\delta _{t_0}) + (1-\rho ^B)f_2^{\langle AB \rangle }(\delta _{t_0}), \end{aligned}$$
which implies the term in (40) is nonpositive, and therefore \({\bar{\rho }} \le \rho ^B \cdot 1 = \rho ^B.\) This completes the proof. \(\square \)
Proof of Theorem 4
The proof of this result follows similar arguments as in the proof of Theorem 3. In particular, see inequality (39). \(\square \)
Proof of Theorem 5
From (7), the necessary and sufficient condition to prefer intervention sequence \(\langle AB \rangle \) rather than \(\langle BA \rangle \) is given by
$$\begin{aligned}&\rho ^A Q^A_E\big (t_0+d^A\big ) + (1-\rho ^A) \rho ^B Q^B_E\big (t_0+d^A+d^B\big ) \nonumber \\&\quad +\, r(A,\delta _{t_0}) + (1-\rho ^A)\sum _{\delta '=1}^{D} P^{(d^A)}(\delta '|\delta _{t_0})r(B,\delta ') \nonumber \\&\ge \rho ^B Q^B_E\big (t_0+d^B\big ) + (1-\rho ^B) \rho ^A Q^A_E\big (t_0+d^A+d^B\big )\nonumber \\&\quad +\, r(B,\delta _{t_0}) + (1-\rho ^B)\sum _{\delta '=1}^{D} P^{(d^B)}(\delta '|\delta _{t_0})r(A,\delta '). \end{aligned}$$
(43)
Since \(q^A(\delta ) = \mu q^B(\delta )\) for all \(\delta \in \Delta \), and \(Q^A_E(t) = \mu Q^B_E(t)\) for all t, we can rearrange the terms in (43), and have
$$\begin{aligned} \mu \ge \frac{\rho ^B Q^B_E\big (t_0+d^B\big ) - (1-\rho ^A)\rho ^B Q^B_E(t_1) + r(B,\delta _{t_0}) - (1-\rho ^A) \sum \nolimits _{\delta =1}^{D}P^{(d^A)} (\delta |\delta _{t_0}) \cdot r(B,\delta ) }{\rho ^A Q^B_E\big (t_0+d^A\big ) + \sum \nolimits _{j=1}^{d^A} \sum \nolimits _{\delta '=1}^{D} P^{(j)}(\delta '|\delta _{t_0}) q^{B}(\delta ') - (1-\rho ^B)\bigg (\rho ^A Q^B_E(t_1) + \sum \nolimits _{j=d^B+1}^{d^A+d^B} \sum \nolimits _{\delta '=1}^{D} P^{(j)}(\delta '|\delta _{t_0}) q^{B}(\delta ')\bigg )}, \end{aligned}$$
(44)
where \(t_1 = t_0 + d^A + d^B\).
Next, we show that \(\mu \le 1\) (otherwise, this result is equivalent to Theorem 3). First, we can verify that both numerator and denominator in (44) are positive. First, define
$$\begin{aligned} g_1&= \rho _A Q^B_E\big (t_0+d^A\big ) - \rho _B Q^B_E\big (t_0+d^B\big ) - (\rho ^A-\rho ^B)Q^B_E\big (t_0 + d^A+d^B\big ), \\ g_2&= - \rho ^A \sum \limits _{j=d^A+1}^{d^A+d^B} \sum \limits _{\delta '=1}^{D} P^{(j)}(\delta '|\delta _{t_0}) q^{B}(\delta ') + \rho ^B \sum \limits _{j=d^B+1}^{d^A+d^B} \sum \limits _{\delta '=1}^{D} P^{(j)}(\delta '|\delta _{t_0}) q^{B}(\delta '). \end{aligned}$$
Next, we verify that \(g_1 + g_2\) is the difference between the denominator and numerator of (44). Observe that the part corresponding to \(g_1\) is straightforward, while \(g_2\) is derived in the following way
$$\begin{aligned}&\sum \limits _{j=1}^{d^A} \sum \limits _{\delta '=1}^{D} P^{(j)}(\delta '|\delta _{t_0}) q^{B}(\delta ') - r(B,\delta _{t_0}) - (1-\rho ^B)\sum \limits _{j=d^B+1}^{d^A+d^B} \sum \limits _{\delta '=1}^{D} P^{(j)}(\delta '|\delta _{t_0}) q^{B}(\delta ') \nonumber \\&\quad + \,(1-\rho ^A)\sum \limits _{\delta =1}^{D}P^{(d^A)} (\delta |\delta _{t_0}) \cdot r(B,\delta ) \nonumber \\&= \sum \limits _{j=1}^{d^A} \sum \limits _{\delta '=1}^{D} P^{(j)}(\delta '|\delta _{t_0}) q^{B}(\delta ') - r(B,\delta _{t_0}) - \sum \limits _{j=d^B+1}^{d^A+d^B} \sum \limits _{\delta '=1}^{D} P^{(j)}(\delta '|\delta _{t_0}) q^{B}(\delta ')\nonumber \\&\quad +\, \sum \limits _{\delta =1}^{D}P^{(d^A)} (\delta |\delta _{t_0}) \cdot r(B,\delta ) \nonumber \\&\quad +\, \rho ^B\sum \limits _{j=d^B+1}^{d^A+d^B} \sum \limits _{\delta '=1}^{D} P^{(j)}(\delta '|\delta _{t_0}) q^{B}(\delta ') - \rho ^A\sum \limits _{\delta =1}^{D}P^{(d^A)} (\delta |\delta _{t_0}) \cdot r(B,\delta ) \nonumber \\&= \bigg (\sum \limits _{j=1}^{d^A} \sum \limits _{\delta '=1}^{D} P^{(j)}(\delta '|\delta _{t_0}) q^{B}(\delta ') - \sum \limits _{j=1}^{d^B} \sum \limits _{\delta '=1}^{D} P^{(j)}(\delta '|\delta _{t_0}) q^{B}(\delta ')\bigg ) \nonumber \\&\quad - \bigg (\sum \limits _{j=d^B+1}^{d^A+d^B} \sum \limits _{\delta '=1}^{D} P^{(j)}(\delta '|\delta _{t_0}) q^{B}(\delta ') - \sum \limits _{j=d^A+1}^{d^A+d^B} \sum \limits _{\delta =1}^{D}P^{(d^A)} (\delta |\delta _{t_0}) \cdot r(B,\delta )\bigg ) + g_2 \nonumber \\&= -\sum \limits _{j=d^A+1}^{d^B} \sum \limits _{\delta '=1}^{D} P^{(j)}(\delta '|\delta _{t_0}) q^{B}(\delta ') + \sum \limits _{j=d^A+1}^{d^B} \sum \limits _{\delta '=1}^{D} P^{(j)}(\delta '|\delta _{t_0}) q^{B}(\delta ') + g_2 = g_2. \end{aligned}$$
(45)
To prove \(g_1 + g_2 \ge 0\), note first
$$\begin{aligned} Q^B_E\big (t_0+d^A\big ) - Q^B_E\big (t_0+d^B\big ) \ge \max _\delta q^B(\delta )\cdot (d^B-d^A) \ge \sum \limits _{j=d^A+1}^{d^B} \sum \limits _{\delta '=1}^{D} P^{(j)}(\delta '|\delta _{t_0}) q^{B}(\delta '), \end{aligned}$$
(46)
$$\begin{aligned} Q^B_E\big (t_0+d^B\big )-Q^B_E\big (t_0 + d^A+d^B\big ) \ge \max _\delta q^B(\delta )\cdot d^A \ge \sum \limits _{j=d^B+1}^{d^A+d^B} \sum \limits _{\delta '=1}^{D} P^{(j)}(\delta '|\delta _{t_0}) q^{B}(\delta '). \end{aligned}$$
(47)
By rearranging the terms in \(g_1\), we have
$$\begin{aligned} g_1&= \rho _A Q^B_E\big (t_0+d^A\big ) - \rho _B Q^B_E\big (t_0+d^B\big ) - (\rho ^A-\rho ^B)Q^B_E\big (t_0 + d^A+d^B\big ) \nonumber \\&= \rho _A \bigg (Q^B_E\big (t_0+d^A\big ) - Q^B_E\big (t_0+d^B\big )\bigg ) \nonumber \\&\quad +\, (\rho ^A-\rho ^B)\bigg (Q^B_E\big (t_0+d^B\big )-Q^B_E\big (t_0 + d^A+d^B\big )\bigg ). \end{aligned}$$
(48)
We also have
$$\begin{aligned} g_2&= - \rho ^A \sum \limits _{j=d^A+1}^{d^B} \sum \limits _{\delta '=1}^{D} P^{(j)}(\delta '|\delta _{t_0}) q^{B}(\delta ') - (\rho ^A-\rho ^B) \sum \limits _{j=d^B+1}^{d^A+d^B} \sum \limits _{\delta '=1}^{D} P^{(j)}(\delta '|\delta _{t_0}) q^{B}(\delta '). \end{aligned}$$
(49)
From (46)–(49), we can verify that \(g_1 + g_2 \ge 0\). Therefore, \(\mu \le 1\). \(\square \)
Proof of Proposition 1
Define \(W^{\langle AB \rangle }(d^A)\) and \(W^{\langle BA \rangle }(d^A)\) to be the reward gained by implementing sequences \(\langle AB \rangle \) and \(\langle BA \rangle \), respectively. We first show that both of the reward is nonincreasing in \(d^A\). Note that
$$\begin{aligned}&W^{\langle AB \rangle }(d^A) \\&= r(A,\delta _{t_0})+\rho ^A Q^A_E\big (t_0+d^A\big )+(1-\rho ^A) \sum _{\delta =1}^{D} P^{(d^A)}(\delta |\delta _{t_0})\cdot r(B,\delta ) \\&\quad +\, (1-\rho ^A)\rho ^B Q^B_E\big (t_0+d^A+d^B\big ) \\&\quad + (1-\rho ^A)(1-\rho ^B)\sum _{\delta =1}^{D} P^{(d^A+d^B)}(\delta |\delta _{t_0})\cdot Q^{\theta ^M}\big (\delta ,t_0+d^A+d^B\big ) \\&= \sum _{j=1}^{d^A} \sum _{\delta =1}^{D} P^{(j)}(\delta |\delta _{t_0}) q^{A}(\delta )+\rho ^A Q^A_E\big (t_0+d^A\big )+(1-\rho ^A) \sum _{j=d^A+1}^{d^A+d^B} \sum _{\delta =1}^{D} P^{(j)}(\delta |\delta _{t_0}) q^B(\delta ) \\&\quad +\, (1-\rho ^A)\rho ^B Q^B_E\big (t_0+d^A+d^B\big ) + (1-\rho ^A)(1-\rho ^B)\sum _{\delta =1}^{D} P^{(d^A+d^B)}(\delta |\delta _{t_0})\\&\quad \times Q^{\theta ^M}\big (\delta ,t_0+d^A+d^B\big ). \end{aligned}$$
Hence,
$$\begin{aligned}&W^{\langle AB \rangle }(d^A) - W^{\langle AB \rangle }(d^A+1 ) \\&= -\sum _{\delta =1}^{D} P^{(d^A+1)}(\delta |\delta _{t_0}) q^{A}(\delta ) + \rho ^A \bigg (Q^A_E\big (t_0+d^A\big ) - Q^A_E\big (t_0+d^A+1\big )\bigg ) \\&\quad +\, (1-\rho ^A) \bigg (\sum _{\delta =1}^{D} P^{(d^A+1)}(\delta |\delta _{t_0}) q^B(\delta ) - \sum _{\delta =1}^{D} P^{(d^A+d^B+1)}(\delta |\delta _{t_0}) q^B(\delta )\bigg ) \\&\quad +\, (1-\rho ^A)\rho ^B \bigg (Q^B_E\big (t_0+d^A+d^B\big )-Q^B_E\big (t_0+d^A+d^B+1\big )\bigg ) + (1-\rho ^A)(1-\rho ^B) \\&\quad \times \,\bigg (\sum _{\delta =1}^{D} P^{(d^A+d^B)}(\delta |\delta _{t_0})\cdot Q^{\theta ^M}\big (\delta ,t_0+d^A+d^B\big ) - \sum _{\delta =1}^{D} P^{(d^A+d^B+1)}(\delta |\delta _{t_0})\\&\quad \times \, Q^{\theta ^M}\big (\delta ,t_0+d^A+d^B+1\big )\bigg ). \end{aligned}$$
Because \(\rho ^A \ge \rho ^B\) and \(Q^A_E(t) - Q^A_E(t+k) \ge Q^B_E(t) - Q^B_E(t+k)\), we have
$$\begin{aligned} \rho ^A \bigg (Q^A_E\big (t_0+d^A\big ) - Q^A_E\big (t_0+d^A+1\big )\bigg ) \ge \rho ^B \bigg (Q^B_E\big (t_0+d^A\big ) - Q^B_E\big (t_0+d^A+1\big )\bigg ). \end{aligned}$$
(50)
From (13), Assumption A3 and (50), we can show that
$$\begin{aligned} -\sum _{\delta =1}^{D} P^{(d^A+1)}(\delta |\delta _{t_0}) q^{A}(\delta ) + \rho ^A \bigg (Q^A_E\big (t_0+d^A\big ) - Q^A_E\big (t_0+d^A+1\big )\bigg ) \ge 0. \end{aligned}$$
(51)
From Assumptions A3–A4 and Lemma 5, we have
$$\begin{aligned} \sum _{\delta =1}^{D} P^{(d^A+1)}(\delta |\delta _{t_0}) q^B(\delta ) - \sum _{\delta =1}^{D} P^{(d^A+d^B+1)}(\delta |\delta _{t_0}) q^B(\delta ) \ge 0. \end{aligned}$$
(52)
Based on Assumption A2, we have that \(Q^B_E\big (t_0+d^A+d^B\big )-Q^B_E\big (t_0+d^A+d^B+1\big ) \ge 0\). Furthermore, Assumptions A4–A5 and Lemma 5 imply that
$$\begin{aligned}&\sum _{\delta =1}^{D} P^{(d^A+d^B)}(\delta |\delta _{t_0})\cdot Q^{\theta ^M}\left( \delta ,t_0+d^A+d^B\right) - \sum _{\delta =1}^{D} P^{(d^A+d^B+1)}(\delta |\delta _{t_0})\nonumber \\&\quad \times Q^{\theta ^M}\left( \delta ,t_0+d^A+d^B+1\right) \ge 0. \end{aligned}$$
(53)
From (51)–(53) we conclude that \(W^{\langle AB \rangle }(d^A) - W^{\langle AB \rangle }(d^A+1 )\) is nonnegative.
Similarly, for intervention sequence \(\langle BA \rangle \), we have
$$\begin{aligned}&W^{\langle BA \rangle }(d^A) \\&= r(B,\delta _{t_0})+\rho ^B Q^B_E\left( t_0+d^B\right) +(1-\rho ^B) \sum _{\delta =1}^{D} P^{(d^B)}(\delta |\delta _{t_0})\cdot r(A,\delta ) \\&\quad +\, (1-\rho ^B)\rho ^A Q^A_E\left( t_0+d^A+d^B\right) + (1-\rho ^A)(1-\rho ^B)\sum _{\delta =1}^{D} P^{(d^A+d^B)}(\delta |\delta _{t_0})\\&\quad \times Q^{\theta ^M}\left( \delta ,t_0+d^A+d^B\right) \\&= \sum _{j=1}^{d^B} \sum _{\delta =1}^{D} P^{(j)}(\delta |\delta _{t_0}) q^{B}(\delta )+\rho ^B Q^B_E\left( t_0+d^B\right) +(1-\rho ^B) \sum _{j=d^B+1}^{d^A+d^B} \sum _{\delta =1}^{D} P^{(j)}(\delta |\delta _{t_0}) q^A(\delta ) \\&\quad +\, (1-\rho ^B)\rho ^A Q^A_E\left( t_0+d^A+d^B\right) + (1-\rho ^A)(1-\rho ^B)\sum _{\delta =1}^{D} P^{(d^A+d^B)}(\delta |\delta _{t_0})\\&\quad \times Q^{\theta ^M}\left( \delta ,t_0+d^A+d^B\right) . \end{aligned}$$
Therefore,
$$\begin{aligned}&W^{\langle BA \rangle }(d^A) - W^{\langle BA \rangle }(d^A+1 ) \\&\quad = -(1-\rho ^B)\sum _{\delta =1}^{D} P^{(d^A+d^B+1)}(\delta |\delta _{t_0}) q^{A}(\delta ) \\&\qquad + (1-\rho ^B)\rho ^A \bigg (Q^A_E\left( t_0+d^A+d^B\right) -Q^A_E\left( t_0+d^A+d^B+1\right) \bigg ) + (1-\rho ^A)(1-\rho ^B)\\&\qquad \times \bigg (\sum _{\delta =1}^{D} P^{(d^A+d^B)}(\delta |\delta _{t_0})\cdot Q^{\theta ^M}\left( \delta ,t_0+d^A+d^B\right) - \sum _{\delta =1}^{D} P^{(d^A+d^B+1)}(\delta |\delta _{t_0})\\&\qquad \times Q^{\theta ^M}\left( \delta ,t_0+d^A+d^B+1\right) \bigg ). \end{aligned}$$
Similar to (51), we have
$$\begin{aligned} -\sum _{\delta =1}^{D} P^{(d^A+d^B+1)}(\delta |\delta _{t_0}) q^{A}(\delta ) + \rho ^A \bigg (Q^A_E\big (t_0+d^A+d^B\big )-Q^A_E\big (t_0+d^A+d^B+1\big )\bigg ) \ge 0. \end{aligned}$$
(54)
Based on (53), we conclude that \(W^{\langle BA \rangle }(d^A) - W^{\langle BA \rangle }(d^A+1 )\) is also nonnegative. Therefore, both \(W^{\langle AB \rangle }(d^A)\) and \(W^{\langle BA \rangle }(d^A)\) are nonincreasing in \(d^A\).
Next, we prove that for any \(d^A\)
$$\begin{aligned} W^{\langle AB \rangle }(d^A) - W^{\langle AB \rangle }(d^A+1 ) \ge W^{\langle BA \rangle }(d^A) - W^{\langle BA \rangle }(d^A+1). \end{aligned}$$
In order to show this fact, we use the following derivations:
$$\begin{aligned}&W^{\langle AB \rangle }(d^A) - W^{\langle AB \rangle }(d^A+1 ) - \bigg (W^{\langle BA \rangle }(d^A) - W^{\langle BA \rangle }(d^A+1)\bigg )\\&\quad = -\sum _{\delta =1}^{D} P^{(d^A+1)}(\delta |\delta _{t_0}) q^{A}(\delta ) + \rho ^A \bigg (Q^A_E\big (t_0+d^A\big ) - Q^A_E\big (t_0+d^A+1\big )\bigg ) \\&\qquad +\, (1-\rho ^A) \bigg (\sum _{\delta =1}^{D} P^{(d^A+1)}(\delta |\delta _{t_0}) q^B(\delta ) - \sum _{\delta =1}^{D} P^{(d^A+d^B+1)}(\delta |\delta _{t_0}) q^B(\delta )\bigg )\\&\qquad +\, (1-\rho ^A)\rho ^B \bigg (Q^B_E\big (t_0+d^A+d^B\big )-Q^B_E\big (t_0+d^A+d^B+1\big )\bigg ) \\&\qquad +\, (1-\rho ^B)\sum _{\delta =1}^{D} P^{(d^A+d^B+1)}(\delta |\delta _{t_0}) q^{A}(\delta ) \\&\qquad - (1-\rho ^B)\rho ^A \bigg (Q^A_E\big (t_0+d^A+d^B\big )-Q^A_E\big (t_0+d^A+d^B+1\big )\bigg ) \\&\quad \ge -\sum _{\delta =1}^{D} P^{(d^A+1)}(\delta |\delta _{t_0}) q^{A}(\delta ) + \rho ^A \bigg (Q^A_E\big (t_0+d^A\big ) - Q^A_E\big (t_0+d^A+1\big )\bigg ) \\&\qquad +\, (1-\rho ^A)\rho ^B \bigg (Q^B_E\big (t_0+d^A+d^B\big )-Q^B_E\big (t_0+d^A+d^B+1\big )\bigg ) \\&\quad \quad - (1-\rho ^B)\rho ^A \bigg (Q^A_E\big (t_0+d^A+d^B\big )-Q^A_E\big (t_0+d^A+d^B+1\big )\bigg ) \\&\quad \ge -\sum _{\delta =1}^{D} P^{(d^A+1)}(\delta |\delta _{t_0}) q^{A}(\delta ) + \rho ^A \rho ^B \bigg (Q^A_E\big (t_0+d^A+d^B\big )-Q^A_E\big (t_0+d^A+d^B+1\big )\bigg )\\&\quad \quad + \,(1-\rho ^A)\rho ^B \bigg (Q^B_E\big (t_0+d^A+d^B\big )-Q^B_E\big (t_0+d^A+d^B+1\big )\bigg )\\&\quad \ge -\sum _{\delta =1}^{D} P^{(d^A+1)}(\delta |\delta _{t_0}) q^{A}(\delta ) + \rho ^B \bigg (Q^B_E\big (t_0+d^A+d^B\big )-Q^B_E\big (t_0+d^A+d^B+1\big )\bigg ) \ge 0. \end{aligned}$$
The first inequality follows by (52) and the nonnegativity of \(q^A(\delta )\). The second inequality is based on (14). The third inequality is due to our assumption \(Q^A_E(t) - Q^A_E(t+k) \ge Q^B_E(t) - Q^B_E(t+k)\) for all t and \(k > 0\) as well as Assumption A3. Finally, the last inequality holds by (13).
As a result, if \(d^A\) increases by one unit, then the decrease in value function \(W^{\langle AB \rangle }(d^A)\) is no smaller than that in \(W^{\langle BA \rangle }(d^A)\). Recall from Theorem 3 that \(W^{\langle AB \rangle }(1) \ge W^{\langle BA \rangle }(1)\) because \(d^B \ge 1\). Therefore, there exists at most one \(\bar{d^A}\), such that we prefer sequence \(\langle AB \rangle \) if \(d^A \le \bar{d^A}\), and prefer sequence \(\langle BA \rangle \), otherwise. \(\square \)