Partial Information Games and Competitive Advertising

Abstract

We study a competitive advertisement process over a social network. Potential customers are hidden in the network, and multiple content providers (CPs) attempt to contact them through advertising. Any customer chooses one among the contacted CPs. The CPs are unaware of the customers already contacted by their opponents. We consider a generic framework applicable to a variety of such problems, to analyse the resulting stochastic game with partial, asymmetric, and non-classical information. Our approach is to consider special strategies, named open-loop control till information update (to manage belief updates). While deriving the best responses, we solve a bi-level control problem; every stage of the dynamic programming equation of the relevant Markov decision process is solved using optimal control tools. We finally reduce the game to a finite state game with finite-dimensional actions, prove the existence of Nash equilibrium and provide structural results. Further, we solve another partial-information game related to project management.

Appendices

Appendix A: Project Completion in Multiple Phases (PCM)

In PCM problem, n agents are competing to win a project, by acquiring M locks before the deadline T. The locks are ordered, and the first one to contact the first lock acquires the project; others lose the project completely. Basically, all the agents compete for the first lock in the beginning, after which the successful agent will attempt for subsequent locks in a given order. One can view it as acquiring and completing M-phases of a project. We say kth contact (of kth lock) is successful if this contact happens before time T and if the agent was the first one to contact all the previous ($k-1$) locks along with the kth lock. The successful agent has the potential to derive subsequent rewards depending upon the number of locks acquired before T hence after.

The contact process of any agent is modelled by independent Poisson process; the rate of the process is chosen by the agent and it can possibly be non-homogeneous. A higher rate increases the probability of earlier contact, but incurs higher cost. After first contact, the successful agent controls the rate to contact further locks, as every lock is associated with a specific reward.

The information structure of this problem is partial and asymmetric. Upon contact the agents get partial information about others; they get to know whether they are the first one to contact; until first contact, they have no information about others’ progress, and may continue even if the lock is already taken by an opponent. As already mentioned, we presented a brief study about this problem in [20] with partial proofs. We now complete the proof, by modelling PCM using the framework of Sect. 3.

The update in information occurs only at the contact epochs, and agents can choose a new contact process for the remaining time interval (i.e. upto time T) based on the updated information. Hence the contact epochs become the natural decision epochs which can be asynchronous across the agents as in Sect. 3. The state of any agent i is the information available to it represented by $z^i_k=(l^i_k,\tau _{k-1}^i)$, where for $k\ge 2$, $\tau _{k-1}^i$ is the $(k-1)$th contact epoch and $l^i_k\in {\mathcal {L}}_k=\{0,1\}$ with 0 representing the failure and 1 the success of the $(k-1)$th contact; also ${\mathcal {L}}_1=\{1\}$ and $\tau _0^i \equiv 0$, and hence $z^i_1=(1,0)$. This (private) state remains the same between two decision epochs. If the agent acquires the project by successful first contact, then all the subsequent contacts may result in some reward if they happen before T. We then call them successful contacts and set the corresponding $l^i_k=1$, otherwise $l^i_k=0$. The set of actions and strategy for any agent is exactly as in Sect. 3; parameters like $\{\beta ^i\}$ and $\nu$ etc., have the same meaning.

The agents can estimate their expected reward for any given strategy profile of the opponents. For any agent i, the reward upon first contact will be $c^i_1$ if it is the first one to contact, zero otherwise. The chances of acquiring this reward will depend upon the contact instance $\tau ^i_1$ and the strategy profile $\pi _{-i}$ of others. Conditioned on $\tau ^i_1\approx s$, the expected reward of agent i is given by,

$$\begin{aligned} c^i_1 (s;\pi _{-i})=c^i_1 P^i_1(s; \pi _{-i}), \end{aligned}$$

(A1)

where $P^i_1 (s; \pi _{-i})$ is the probability that i is first one to contact the first lock; this is the probability that none of the opponents have contacted the first lock before instance s, under opponent strategy profile $\pi _{-i} = \{a_k^m (\cdot )\}_{m\ne i,k}$. Recall here that, the locks are ordered and that any agent attempts to contact only the first lock (at the start), and thus

$$\begin{aligned} P^i_1 (s; \pi _{-i})= & {} \hbox {e}^{-\sum _{m \ne i} {{{\bar{a}}}}_1^m (s) }, \text{ with } \ {{{\bar{a}}}}_1^m (s) =\int _{\tau ^m_0}^s a^m_1(t) \hbox {d}t, \end{aligned}$$

(A2)

exactly as in (7); observe here that the search is only for one lock, thus the rate of Poisson clock for any agent m is given by $a_1^m (\cdot )$ and hence $q(l_1^m)= 1 = l_1^m$.

If agent i is successful in acquiring the project then $l^i_2 = 1$ [and would have received the reward $c_1^i$ as in (A1)], otherwise $l^i_2 = 0$. When $l_2^i = 1$, it can receive the rewards for subsequent stages if the corresponding locks are acquired before T. Thus for any $k \ge 2$, if $\tau _k^i < T$ then $l_{k+1}^i = 1$ and it would receive reward $c^i_k$. On the other hand, if the agent fails to acquire the project, i.e. $l_2^i=0$, then the agent receives no subsequent rewards and $l_{k+1}^i=0$ for all $k \ge 2$. To model this problem using framework of Sect. 3, we set $q(\cdot )$ and the rewards for $k\ge 2$ conditioned on $\tau ^i_k\approx s$ (see footnote 4), respectively, as below,

$$\begin{aligned} q(l^i_k) = l^i_k \text{, } \text{ and, } ( c^i_k (s;\pi _{-i}),\ l^i_{k+1}):=\left\{ \begin{array}{llll} (c^i_k,\ l^i_k) &{} \text{ if } \quad s\le T,\\ ( 0,\ 0)&{} \text{ else. } \end{array}\right. \end{aligned}$$

(A3)

The rationale behind the above modelling details are as follows: (i) when there is a failure in the previous stage, i.e. when $l^i_k=0$, $q(l^i_k) = 0$ and hence the Poisson density is zero till T (see footnote 4), thus the corresponding $\tau _k^i > T$ and hence the reward of such a stage is zero; (ii) with successful $(k-1)$th stage, i.e. with $l^i_k = 1$, the agent attempts and gets a reward $c_k^i$ only if $\tau _k^i < T$, which in turn depends upon the action $a_k^i$. Hence, the expected reward conditioned on state $z_k^i$ equals zero if $l^i_k=0$ and otherwise is given by the following [using (7) and (12)],

$$\begin{aligned} E_{ z_k^i }[c^i (T\wedge \tau _k^i );\pi _{-i} ]= & {} \int _{\tau ^i_{k-1}}^{T} c^i(s;\pi _{-i})a^i_k(s) \hbox {e}^{-{\bar{a}}^i_k(s)} \hbox {d}s. \end{aligned}$$

(A4)

The PCM problem can thus be formulated as in (8) with $r^i_k$ defined exactly as in (9) and (12) using (A1)–(A4). For clarity, we summarize the modelling details at one place:

• At the start, every agent searches for the first lock, and, hence $l_1^i = 1$ and $q(l_1^i) = l_1^i = 1$ for all i.

• At maximum, only one agent (say agent i) can have $l_2^i = 1$, while for the rest $l_2^m = 0$, when $m \ne i$. In fact $l_k^m = 0$ for all $k \ge 2$ for such agents.

• For successful agent i, it is possible that $l_k^i = 0$ for some $k > 2$, call it ${{{\bar{k}}}}$. This happens when the corresponding $\tau _{{{{\bar{k}}}}-1}^i > T$. Also for further $k \ge {{{\bar{k}}}}$, we have $l_k^i = 0$.

• When $l^i_k = 0$ (i.e. when $(k-1)$th stage is a failure), $q(l^i_k)=l^i_k = 0$ captures the fact that the agent stops further attempts after one failure.

We now prove the required assumptions ${\mathbb {A}}$.1 and ${\mathbb {A}}$.2, so as to utilize the results of Sect. 3. For ${\mathbb {A}}$.1, the transition probability for any $t \le T$ equals (set $z^i_0=(l^i_1,\tau ^i_0)=(1,0)$),

$$\begin{aligned} P_k(l^i_{k+1} \mid l^i_k,t,\pi _{-i})= & {} {\left\{ \begin{array}{ll} P^i_1 (t; \pi _{-i}) &{} \text { if } k = 1 \text { and } l^i_{k+1}=1,\\ 1-P^i_1 (t; \pi _{-i}) &{}\text { if } k=1 \text { and } l^i_{k+1}=0,\\ \mathcal{X}_{ \left\{ l^i_{k+1}=l^i_k\right\} } &{} \text { if } k\ge 2, \end{array}\right. } \end{aligned}$$

which is clearly Lipschitz continuous in t for any $l^i_{k+1}, l^i_k, \pi _{-i}$. Recall ${\mathcal {L}}_1=\{1\}$ (a singleton), and observe $P^i_1(\cdot ;\pi _{-i})$ is a monotone function which implies the required stochastic dominance of ${\mathbb {A}}$.1 for $k=1$. For $k\ge 2$, and any monotone function $b(\cdot )$, from the third row of the above transition probability, we have $E[b(l^i_{k+1}) \mid l^i_k,\cdot ,\cdot ]=b(l^i_k)$; thus ${\mathbb {A}}$.1 is satisfied for all k. From equations (A1)–(A3) the reward functions $\{c^i_k (\cdot ;\pi _{-i})\}_k$ are non-increasing and Lipschitz continuous for any $\pi _{-i}$, satisfying ${\mathbb {A}}$.2.

Therefore, Theorem 1 is applicable. This completes the proof of the statement that reduced game with MT-strategies can provide NE for the original game corresponding to PCM. Hence, one can find a NE (if it exists) in the reduced game.

We in fact have unique NE in this reduced game, as discussed in [20]. This is proved by further reducing the game to one-dimensional game (see [20]). The uniqueness of NE among MT-strategies, and a method to find the unique NE is provided in [20, Theorems 6–7]. We reproduce the result here for the sake of completion (see [20, 21] for proof):

Theorem 5

The unique NE is given by the set of thresholds $\{\theta _k^{i*}\}_{i \le n, k\le M}$ defined recursively along with ‘NE-value functions’ $\left\{ \Upsilon _{k}^{i*} (\cdot )\right\}$ as below for any $i\le n$ and $k\le M$: (i) for $k=M$, i.e. for the last lock we have,

$$\begin{aligned} \theta _M^{i*} = T \mathcal{X}_{ \left\{ c^i_M> \nu \right\} } \text{ and } { \Upsilon }_{M}^{i*} (t):= (c^i_M-\nu ) \left( 1 - \hbox {e}^{- \beta ^i (T-t) } \right) \mathcal{X}_{ \left\{ c^i_M > \nu \right\} }, \end{aligned}$$

(ii) for any $2\le k < M (\text{ with } \emptyset \text{- } \text{ null } \text{ set})$

$$\begin{aligned}{} & {} \theta _{k}^{i*} \,= \ \inf \{ t \ge 0: c^i_{k}+ { \Upsilon }_{k+1}^{i*} (t) \le \nu \}, \text{ with, } \inf \emptyset := T, \\ { \Upsilon }_k^{i*}(t) = \mathcal{X}_{ \left\{ t < \theta ^{i*}_k \right\} } \int _{t}^{ \theta ^{i*}_k } (c^i_k+ & {} {\Upsilon }_{k+1}^{i*} (s)-\nu )\beta ^i \hbox {e}^{-\beta ^i (s-t) }\hbox {d}s, \text{ and, } \end{aligned}$$

(iii) for $k=1$ (for the first lock), thresholds simultaneously satisfy:

$$\begin{aligned} \theta _1^{i*} = \inf \left\{ t: ( \Upsilon _2^{i*} (t) + c^i_1 ) \hbox {e}^{ - \sum _{m \ne i} \beta ^m ( t \wedge \theta ^{m*}_1 )} \le \nu \right\} \wedge T, \text{ for } \text{ all } 1\le i\le n. \end{aligned}$$

(A5)

$\square$

Remarks We found NE of PCM game by reducing it to a much simpler game such that the NE of the reduced game would also be the NE of the original game. Theorem 5 gives a procedure to find the unique NE of the reduced game. In [20], we computed approximate closed-form expressions for NE for some cases. We have also provided some interesting numerical examples in [20] and some more in [21]. Some of the important conclusions related to this application are: (a) one either tries the last lock till the end or never attempts it, i.e. $\theta ^{i*}_M \in \{0,T\}$, (b) the thresholds can be non-monotone (in k) depending upon rewards associated with various stages (locks acquired). The non-monotonicity is observed if the rewards in later stages (after more lock acquisitions) are much higher than those in the earlier stages; then the optimal policy at some time instances allows the agent to continue with further acquisition process only if the number of acquisitions is more (see figures 2–4 in [21]).

Appendix B: Proofs Related to Theorem 1

Proof of Theorem 2:

The control problem (18) and (19) for any k can be rewritten as:

$$\begin{aligned} u(s,{{y}})= & {} \sup _{ a (\cdot ) } \left\{ \int _s^T L_k(s', x (s'), a(s') ) \hbox {d}s' + g(x (T) ) \right\} \text{ with } \\ L_k(s, {{y}}, a):= & {} \Big ( c^i_k(s;\pi _{-i}) + \hspace{-3mm}\sum _{l_{k+1}\in {\mathcal {L}}_{k+1}}\hspace{-3mm} v_{k+1}^i ((l_{k+1},s);\pi _{-i}) P(l_{k+1} \mid l_k,s,\pi _{-i}) \\{} & {} - \nu {{y}} \Big ) f_{q(l_k)}(s) \hbox {d}s, \end{aligned}$$

and $g({{y}})= - \nu {{y}} {{\hbox {e}^{ -q(l_k)y}}}.$ For $k=M$, $v^i_{M+1}$ is defined to be 0, and

$$\begin{aligned} L_M(s, {{y}}, a)= & {} \left( c^i_M(s;\pi _{-i}) - \nu {{y}} \right) f_{q(l_k)}(s). \end{aligned}$$

Observe that ${{y}}$ can be confined to range $[0,\beta T]$. Then $L_M(\cdot )$ is bounded on $[0,T]\times [0,\beta T]\times [0,\beta ]$ and is also Lipschitz continuous by assumption ${\mathbb {A}}$.2. Further clearly the RHS of the ODE (19) and the terminal cost g are all bounded and Lipschitz continuous. Thus, by [12, Theorem 10.1 and the following Remark 10.1, Chapter 2] the value function $u(\cdot , \cdot )$ is unique viscosity solution which is Lipschitz continuous when $k= M$. This implies $v^i_{M}((l, s);\pi _{-i})$ is Lipschitz continuous in s, further it is also bounded [see (17)]. Assume this holds true for $k+1,k+2\dots M$, then for k, from the induction hypothesis, $v_{k+1}$ is Lipschitz continuous and from ${\mathbb {A}}$.1 and ${\mathbb {A}}$.2, $L_k(\cdot )$ is bounded on $[0,T]\times [0,\beta T]\times [0,\beta ]$ and is also Lipschitz continuous. Again, RHS of the ODE (19) and the terminal cost g are all bounded and Lipschitz continuous; again using the results for optimal control problem (17) defining the kth stage DP equation. Thus by the same result, the value function $u(\cdot , \cdot )$ is unique viscosity solution which is Lipschitz continuous. By backward induction on k, part (i) and (ii) are true.

For part (iii), we apply the results of [19]. Towards this, the optimal control problem can be converted into Mayer-type (finite horizon problem with only terminal cost) by usual technique of augmenting a new component to state which represents

$$ {\tilde{y}}(s'):= \int _{s}^{s'} L_k( {{{\tilde{s}}}}, x({{{\tilde{s}}}}), a ({{{\tilde{s}}}}) ) \hbox {d}{{{\tilde{s}}}}, $$

and equivalently maximizing ${\tilde{y}}(T) + g(x(T) )$. By part (ii) all the required assumptions [19, Assumptions (i) to (vii)] are satisfied, with compact control space $U = [0,\beta ]$, compact state space ${{{\hat{X}}}} = [0, \beta T]$ (it is easy to verify that the state variable could be confined to this range); assumptions (i)–(ii) are trivially satisfied; assumption (iii) follows by part (i); for assumption (iv) one can actually bound by a constant independent of $(s', ({{y}},{\tilde{y}}))$; easy to verify convexity requirement of (vii), because for any given $(s', ({{y}},{\tilde{y}}))$ the set in mention is an interval. This proves part (iii). $\square$

Proof of Lemma 1:

For any stage k, given any state $z^i_k$, consider the open-loop policy $a_k^i(\cdot )$, briefly represented as $a(\cdot )$. If $a(\cdot )$ is already a threshold type, we have nothing to prove. If not, choose two intervals $[t_1,t_1+\delta _1]$ and $[t_2,t_2+\delta _2]$ with $t_2 \ge t_1+\delta _1$ such that $\int _{t_1}^{t_1+\delta _1}a(t)\hbox {d}t<\beta \delta _1 \text{ and } \int _{t_2}^{t_2+\delta _2}a(t)\hbox {d}t>0.$ Further by choosing appropriate end points, $\int _{t_1}^{t_1+\delta _1}a(t)\hbox {d}t + \int _{t_2}^{t_2+\delta _2}a(t)\hbox {d}t=\beta \delta _1.$

Now construct policy ${a'}(t)$ such that, $\int _{t_1}^{t_1+\delta _1}{a'}(t)\hbox {d}t=\beta \delta _1$ and $\int _{t_2}^{t_2+\delta _2}{a'}(t)\hbox {d}t=0,$ and on rest of the intervals the policy ${a'}(\cdot )$ matches completely with $a(\cdot )$. This new policy is basically constructed by shifting the mass from a later interval $[t_2,t_2+\delta _2]$ to a former interval $[t_1,t_1+\delta _1]$ in policy $a(\cdot )$, note that if one can’t find such intervals, it implies that the policy $a(\cdot )$ itself is a threshold policy and we have nothing to prove.

Observe for all $t < t_1$ we have, $a(t) = {a'}(t) \text{ and } \text{ hence } {\bar{a}}(t) =\int _{\tau _{k-1}}^t a (s) \hbox {d}s = {\bar{a}}'(t).$ Using similar logic, for any $t\in ( t_1, t_2]$, ${\bar{a}}(t) \le {\bar{a}}'(t)$; for any $t\in ( t_2, t_2 + \delta _2 ]$ we have ${\bar{a}}(t) \le {\bar{a}}'(t)$; and for all $t > t_2 + \delta _2$ we have ${\bar{a}}(t) = {\bar{a}}'(t)$.

This implies, the time to transition to next stage with policy $a(\cdot )$ denoted as $\tau _a$ is stochastically dominated by that with policy $a'(\cdot )$ denoted as $\tau _{a'}$ as explained below; consider the CDFs with both the policies; i) for any $x<t_1$,

$$F_{a}(x) = Prob (\tau _a \le x) = 1- \hbox {e}^{-q(l_k){\bar{a}}(x) } = 1- \hbox {e}^{-q(l_k){\bar{a}}'(x) } =F_{a'}(x),$$

ii) for any $x\in (t_1,t_1+\delta _1)$, we have ${\bar{a}}(x)<{\bar{a}}'(x)$ and so $F_{a}(x)<F_{a'}(x),$ iii) for any $x\in (t_2,t_2+\delta _2),$ we have ${\bar{a}}(x)\le {\bar{a}}'(x)$ and so $F_{a}(x)\le F_{a'}(x),$ and iv) for any $x\in [t_2+\delta _2,T],$ we have ${\bar{a}}(t)={\bar{a}}'(t)$ and so $F_{a}(x)=F_{a'}(x).$ This proves the required stochastic dominance, $\tau _a {\mathop {\le }\limits ^{d}} \tau _{a'}$.

Further observe that the expected cost with policy $a(\cdot )$ (by change of variables and the fact that ${\bar{a}}(\tau _{k-1})=0$),

$$\begin{aligned}{} & {} E[ {{{\bar{a}}}} (\tau _a ); \tau _a < T ] + {{{\bar{a}}}} (T ) \hbox {e}^{-q(l_k){{{\bar{a}}}}(T)}\\{} & {} \quad = \int _{\tau _{k-1}}^{T}\hspace{-3mm} {{{\bar{a}}}} (t) q(l_k) a(t) \hbox {e}^{- q(l_k){{{\bar{a}}}} (t) } \hbox {d}t + {{{\bar{a}}}} (T ) \hbox {e}^{-q(l_k){{{\bar{a}}}}(T)}, \\{} & {} \quad = \int _0^{{{{\bar{a}}}}(T) } q(l_k) x \hbox {e}^{-q(l_k) x} \hbox {d}x + {{{\bar{a}}}} (T ) \hbox {e}^{-q(l_k){{{\bar{a}}}}(T)} =\frac{1}{q(l_k)} \left( 1 - \hbox {e}^{-q(l_k){{{\bar{a}}}}(T)}\right) , \end{aligned}$$

which is the same as that using $a'$ because ${{{\bar{a}}}}(T) = {\bar{a}}'(T)$. One can keep on improving the policy until it becomes a threshold policy. This completes the proof. $\square$

We require the following lemma in the proof of Lemmas 2 and 3.

Lemma 5

Let $J_h(t,y,a )$ be a function of the form

$$\begin{aligned} J_h(t,y,a ) =\int _{t}^{T} \left( h (s) - \nu x (s) \right) q a ( s) \hbox {e}^{-qx(s)}\hbox {d}s - \nu x(T) \hbox {e}^{-qx(T) }, \end{aligned}$$

defined using continuous function $h(\cdot )$ and state process ${\dot{x}} (s) = a ( s),$ with initial condition $x(t)=y.$ Define $u(t,y):= \sup _{\ a \in L^\infty } J_h (t,y,a)$, then we have:

• (i) $J_h(t,y,a )=\hbox {e}^{-qy}[J_h(t,0,a )-\nu y]$,

• (ii) $u(t,y)=\hbox {e}^{-qy}[u(t,0)-\nu y]$,

• (iii) The optimal policy $a^*(\cdot )$ is independent of initial condition y.

Proof

By change of variable $x(s)={{y}}+{\tilde{x}}(s),$ we have ${\mathop {{\tilde{x}}}\limits ^{\varvec{\cdot }}} (s) = a ( s)$, i.e. ${\tilde{x}}(s) = \int _t^s a ({{{\tilde{s}}}}) \hbox {d}{{{\tilde{s}}}}$ with ${\tilde{x}}(t)=0$, and hence,

$$\begin{aligned} J_h(t,{{y}},a )= & {} \int _{t}^{T} \Big ( h (s) - \nu \big ( {{y}}+{\tilde{x}} (s) \big ) \Big ) q a ( s) {{\hbox {e}^{ -q(y+{\tilde{x}}(s))}}} \hbox {d}s\\{} & {} -\nu \left( {{y}}+{\tilde{x}}(T)\big ) {{\hbox {e}^{-q(y+{\tilde{x}}(T))}}}\right) ,\\= & {} {{\hbox {e}^{-qy}}} \Big ( \int _{t}^{T}\hspace{-2mm} \big ( h (s) - \nu {\tilde{x}} (s) \big ) qa ( s)\hbox {e}^{ -q {\tilde{x}}(s) }\hbox {d}s\hspace{-1mm} - \nu {\tilde{x}}(T) \hbox {e}^{-q{\tilde{x}}(T)}\\{} & {} - \nu {{y}} \int _{t}^{T} \hspace{-2mm} q a( s)\hbox {e}^{ - q{\tilde{x}}(s)} \hbox {d}s -\nu {{y}} \hbox {e}^{-q{\tilde{x}}(T)} \Big ),\\= & {} {{\hbox {e}^{-qy}}} \left( J_h(t, 0,a) - \nu {{y}} \right) , \hspace{5mm} \ \ \text{ because, } \ \int _{t}^{T} q a( s)\hbox {e}^{ - q{\tilde{x}}(s)}\hbox {d}s + \hbox {e}^{-q{\tilde{x}}(T)} =1. \end{aligned}$$

This completes part (i). For part (ii), from the above equation we have,

$$\begin{aligned} u(t, {{y}})=\sup _{a\in L^{\infty } } J_h(t, {{y}},a)= & {} \sup _{a\in L^{\infty }} {{\hbox {e}^{-qy}}} \left[ J_h (t, 0,a) - \nu {{y}} \right] ,\\= & {} {{\hbox {e}^{-qy}}}\left[ \sup _{a\in L^{\infty } } J_h (t, 0,a) - \nu {{y}} \right] , \end{aligned}$$

and hence we have, $u(t,{{y}})={{\hbox {e}^{-qy}}}[u(t,0)-\nu {{y}}].$ This proves part (ii). Further it is clear from above that the optimal policy $a^*(\cdot )$ remains the same for all initial conditions y and this proves part (iii).  $\square$

Proof of Lemma 2:

We prove it in two steps. For brevity, we drop $\pi _{-i}$ and i.

Step 1 We prove $v_k(l_k,t)\ge v_k(l_k,\tau )$ if $\tau \ge t$. For any stage k and state $z_k=(l_k,t)$, the kth stage DP equation equals the following optimal control problem [see (18)],

$$\begin{aligned} u(t,0) = \sup _{ \ a (\cdot ) \in L^\infty } \Big \{ \int _{t}^{T} \hspace{-3mm} \big ( h_k (s')- \nu x (s')\big ) q(l_k) a ( s') \hbox {e}^{-q(l_k) x(s')} \hbox {d}s' +g(x(T )) \Big \}. \end{aligned}$$

By dynamic programming principle [12, Theorem 5.1] as applied to this optimal control problem, we can rewrite it as follows:

$$\begin{aligned} u(t,0 )= & {} \hspace{-4mm}\sup _{ a_k \in L^\infty [t, \tau ] }\left\{ \int _{t}^{\tau } \big ( h_k (s)- \nu x (s) \big ) q(l_k) a_k ( s) \hbox {e}^{-q(l_k) x(s)}\hbox {d}s + u(\tau ,x(\tau )) \right\} , \\ \text{ where, }{} & {} u(\tau , x(\tau )) = \sup _{ a_k \in L^\infty [ \tau , T] } J (\tau , x(\tau ), \ a_k ). \end{aligned}$$

Observe that the function $J ( \tau , x(\tau ), a_k)$ [see Eq. (18)] has same structure as function $J_h (t, x, a)$ in Lemma 5 with $h = h_k$, $t = \tau$, $x = x(\tau )$ and $a= a_k$ and continuity follows by Theorem 2. Hence we have,

$$\begin{aligned} u(\tau , x(\tau ) )= & {} \hbox {e}^{-q(l_k)x(\tau )} \left[ u(\tau , 0) - \nu x(\tau ) \right] , \text{ and } \text{ so } \\ u(t,0 )= & {} \hspace{-2mm}\sup _{ a_k\in L^\infty [t, \tau ]}\hspace{-1mm}\Big \{\int _{t}^{\tau } \big (h_k(s) - \nu x (s) \big ) q(l_k){{a_k(s)}} \hbox {e}^{-q(l_k) x(s)}\hbox {d}s \\{} & {} +\, \hbox {e}^{-q(l_k) x(\tau )}[ u(\tau , 0 )-\nu x(\tau )]\Big \}. \end{aligned}$$

In the above under zero policy ($a_k(s)= 0$ for all $s \in [t, \tau ]$), we have $x(\tau )=0$ and the first term (integral) in the above supremum is zero; hence $u(t,0)\ge u(\tau ,0 ).$ Thus, $v_k (z_k) \ge v_k ({\bar{z}}_k),$ as optimal control value $u(\tau , 0)$ corresponds to state ${\bar{z}}_k=(l_k,\tau )$.

Step 2 We prove $v^i_k((l,t);\pi _{-i})\ge v^i_k((l',t);\pi _{-i})$ if $l\ge l'$. Let $q(l)=q$, $q(l')=q'$ and respective transition times, $\Omega \sim f_q$ and $\Omega ' \sim f_{q'}$ etc. By ${\mathbb {A}}$.1, $\Omega {\ {\mathop {\le }\limits ^{d}} \ } \Omega '$. Begin backward induction with M, $z_M=(l,t)$ and $z'_M=(l',t)$. For any policy a, from (15) and (16),

$$\begin{aligned} J_M(z_M,a)= & {} \int _t^T( c_M(s)-\nu {\bar{a}}(s))f_{q}(s) \hbox {d}s- \nu {\bar{a}}(T)\hbox {e}^{-q {\bar{a}}(T)},\\\ge & {} \int _t^T( c_M(s)-\nu {\bar{a}}(s))f_{q'}(s) \hbox {d}s- \nu {\bar{a}}(T)\hbox {e}^{-q' {\bar{a}}(T)} = J_M(z_M',a), \end{aligned}$$

where the inequality of the first term follows because of $\Omega {\ {\mathop {\le }\limits ^{d}} \ } \Omega '$ and monotonicity of $c_M(\cdot )-\nu {\bar{a}}(\cdot )$ and that of the second term follows since $q\ge q'$ again by ${\mathbb {A}}$.1. Thus from (14), $v_M(l,t)\ge v_M(l',t)$. Assume it is true for $M\dots k+1$, and consider k, $z_k=(l,t)$ and $z_k'=(l',t)$. Fix policy a, then $\sum _{{\tilde{l}}} v_{k+1} ({\tilde{l}},s) P({\tilde{l}} \mid l,s)\ge \sum _{{\tilde{l}}} v_{k+1} ({\tilde{l}},s) P({\tilde{l}} \mid l',s)$ for any s by ${\mathbb {A}}$.1 and induction. Thus, from (15) and (16) (since $\Omega {\ {\mathop {\le }\limits ^{d}} \ } \Omega '$),

$$\begin{aligned} J_k(z_k,a)\ge & {} \int _t^T\hspace{-3mm}\Big ( c_k(s)+\sum _{{\tilde{l}}\in {\mathcal {L}}_{k+1}} v_{k+1} ({\tilde{l}},s) P({\tilde{l}} \mid l',s) -\nu {\bar{a}}(s)\Big )f_{q}(s) \hbox {d}s- \nu {\bar{a}}(T)\hbox {e}^{-q {\bar{a}}(T)},\\\ge & {} \int _t^T\Big ( c_k(s)+\sum _{{\tilde{l}}\in {\mathcal {L}}_{k+1}} v_{k+1} ({\tilde{l}},s) P({\tilde{l}} \mid l',s) -\nu {\bar{a}}(s)\Big )f_{q'}(s) \hbox {d}s- \nu {\bar{a}}(T)\hbox {e}^{-q'{\bar{a}}(T)},\\= & {} J_k(z_k',a). \end{aligned}$$

Thus from (14), $v_k(l,t)\ge v_k(l',t)$. Combining the two steps, we have the result. $\square$

Appendix C: Proofs Related to Sect. 4

Lemma 6

The probability $P^i_m(s;\pi _{-i})$ is Lipshcitz continuous in s.

Proof

Define $Q^{j}(s;\pi ^j)$ to be the probability that agent j (among opponents) has contacted the particular lock l till time s using its own strategy $\pi ^j$ contained in $\pi _{-i}$. Then,

$$\begin{aligned} P^i_m(s;\pi _{-i})= & {} \sum _{{\mathcal {J}}: \mid {\mathcal {J}} \mid =m-1, i \not \in {\mathcal {J}}}\left( \prod _{j\in {\mathcal {J}}} Q^{j}(s;\pi ^j)\right) \prod _{j\not \in {\mathcal {J}},j\ne i}\left( 1- Q^{j}(s;\pi ^j)\right) \end{aligned}$$

(C6)

Observe for any agent j,

$$\begin{aligned} Q^j(s;\pi ^j)= & {} \sum _{k=1}^{M}\frac{k}{M}Q^j(s;k,\pi ^j), \end{aligned}$$

(C7)

where k/M is the probability that l is among the contacted k locks and $Q^j(s;k,\pi ^j)$ is the probability that j has contacted k locks by time s, and is given by:

$$\begin{aligned} Q^j(s;k,\pi ^j)= \int _0^{s} f_M(t_1)\int _{t_1}^{s} \dots \int _{t_{k-1}}^{s}f_{M-k+1}(t_k)\hbox {e}^{-(M-k){\bar{a}}^j_{k+1}(s)}\hbox {d}t_k\dots \hbox {d}t_1, \end{aligned}$$

(C8)

where $f_{M-l+1}(\cdot ):=(M-l+1) a^j_{l}(\cdot ) \hbox {e}^{-(M-l+1) {\bar{a}}^j_{l}(\cdot )}$ is the density of contact time. Also ${\bar{a}}^j_{l}(t)=\int _{t_{l-1}}^t a^j_{l}(t') \hbox {d}t'$ [see (7)]. By integral definition given in equations (C6)–(C8) and boundedness (e.g. by $\beta ^j$), the function $P^i_m(\cdot ;\pi _{-i})$ is Lipschitz continuous in s. $\square$

Lemma 7

For any strategy $\Theta _{-i}$ and $\Theta '_{-i}$ of opponents, if time s satisfies the following for some $M'$,

$$\begin{aligned} \min _{j\ne i}\theta ^j_m\ge s \text{ and } \min _{j\ne i}\theta '^j_m\ge s\ \ \text{ for } \text{ all } \text{ locks } m<M', \end{aligned}$$

and $\theta ^j_m=\theta '^j_m$ for all $m\ge M'$, then $c^i(t,\Theta _{-i})=c^i(t,\Theta '_{-i})$ for all $t\le s.$

Proof

From Eq. (C8), the probability that any agent j has contacted k lock by time t using strategy $\Theta _j$ of $\Theta _{-i}$, $Q^j(s;k,\Theta _{-i})$ is same as that using $\Theta '_j$ of $\Theta '_{-i}$ for all $t\le s$. Basically, both the opponent strategies are the same till s; there is a difference only after s. Then $Q^j(t;k,\Theta _{-i})=Q^j(t;k,\Theta '_{-i})\ \ \text{ for } \text{ all } j, \text{ for } \text{ all } t\le s.$ Using (C7), (C6) and (1), we have the result. $\square$

Proof of Lemma 4:

For the ease of notation, in this proof we denote $c^i (t;\Theta _{-i})$ as c(t), and $v_{k}^i ((M-k+1,t);\Theta _{-i})$ as $v_k(t)$. Let $\Omega _k$ represent random variable distributed according to density $f_k$. By DP equations (27) and (28), now with $\tau ^i_{k}=t$, $\forall k$:

$$\begin{aligned} v_{k+1}(t)= & {} \sup _{\theta \in [t,T]}\int _0^{\theta -t}\left( c(s+t)-\nu _{k+1}+v_{k+2}(s+t)\right) f_{M-k} (s) \hbox {d}s, \nonumber \\= & {} \sup _{\theta \in [t,T]} E [ g_{k+1} (\Omega _{M-k}; \theta ) ],\nonumber \\{} & {} \text{ with, } g_k (s; \theta ):= \left( c(s+t)-\nu _{k} +v_{k+1}(s+t) \mathcal{X}_{ \left\{ k<M\right\} } \right) \mathcal{X}_{ \left\{ s < \theta -t\right\} }. \end{aligned}$$

(C9)

Observe for $k=M$,

$$\begin{aligned} v_M(t)= & {} \sup _{\theta \in [t,T]}\int _0^{\theta -t}\left( c(s+t)-\nu _{M}\right) f_1(s) \hbox {d}s, \\= & {} \sup _{\theta \in [t,T]} E[g_M (\Omega _1;\theta )]. \end{aligned}$$

We prove using backward induction. As $v_M (s) \ge 0$, we have for $k=M-1$ and all $s,\theta$, $g_{M-1} (s; \theta )=(c(s+t)-\nu _{M-1} + v_M(s+t)) \mathcal{X}_{ \left\{ s<\theta -t\right\} } \ge (c(s+t)-\nu _{M}) \mathcal{X}_{ \left\{ s<\theta -t\right\} }=g_M (s; \theta )$. Further $\Omega _1{\mathop {\ge }\limits ^{d}}\Omega _2$ (stochastically dominates), and $g_M(\cdot )$ is decreasing by ${\mathbb {A}}$.2 (for inequality a)

$$\begin{aligned} \text{ and } \text{ so, } v_{M-1}(t)&= \sup _{\theta \in [t,T]} E [g_{M-1} (\Omega _{2})] \ge \sup _{\theta \in [t,T]} E [g_{M} (\Omega _{2})],\\&{\mathop {\ge }\limits ^{a}} \sup _{\theta \in [t,T]} E [g_{M} (\Omega _1) ]=v_M(t). \end{aligned}$$

Assume $v_{l+1}(s)\ge v_{l+2}(s)$ for all $s, \ l\ge k$. Then from (C9) $g_{k} (s; \theta ) \ge g_{k+1} (s; \theta )\forall s, \theta ,$

Inequality a is true because $\Omega _{M-k}{\mathop {\ge }\limits ^{d}}\Omega _{M-k+1}$ and function $g_{k+1}(\cdot )$ is decreasing again by ${\mathbb {A}}$.2 ($c(\cdot )$ is decreasing) and also using Lemma 2 ($v_{k+2}(\cdot )$ is decreasing). $\square$

Proof of Theorem 3:

Fix opponents’ MT strategy as $\Theta _{-i}$, let $\nu _{k}:=\frac{\nu }{M-k+1}$. For brevity, here we denote $c^i (t;\Theta _{-i})$ as c(t), $h^i_k (t;\Theta _{-i})$ as $h_k(t)$, $v_{k}^i ((M-k+1,t);\Theta _{-i})$ as $v_k(t)$ and ${\mathcal {B}}_{k}(\Theta _{-i})$ as ${\mathcal {B}}_{k}$. For any k, if $h_k(0) \le \nu _{k}$ then from Lemma 4, $h_{k'} (0) \le \nu _{k'}$ and so ${\mathcal {B}}_{k'}=\{0\}$ from (29) for all $k'\ge k$. Clearly, the value function at end, $v_{k+1}(T)=0$, and so $h_k (T)=c (T)$. Thus, for any k, $c (T)\ge \nu _{k}$ implies $c (T ) \ge \nu _{k'},$ and ${\mathcal {B}}_{k'}=\{T\}$ [from (29)] for all $k'\le k$.

Consider other k, i.e, with $c (T )< \nu _{k}<h_{k} (0)$. Then clearly the BR sets are, ${\mathcal {B}}_k=\left\{ t\ge 0:c(t) + v_{k+1} (t) =\nu _{k} \right\}$ and ${\mathcal {B}}_{k+1}\subset \left\{ t\ge 0: c(t) + v_{k+2} (t) =\nu _{k+1}\right\} \cup \{0\}$. So by monotonicity, for any $t\in {\mathcal {B}}_k$, $c(t)=\nu _{k} -v_{k+1} (t)<\nu _{k+1} -v_{k+2} (t).$ This implies $t'<t$ for any $t\in {\mathcal {B}}_k$ and $t'\in {\mathcal {B}}_{k+1}$ for the above (interior) type of k. Thus in all, any BR strategy $\Theta _i^*$ has $\theta ^{i*}_{k}\ge \theta ^{i*}_{k+1}$, with equality only when $\theta _k^{i*}\in \{0,T\}$. This proves part (i). Further by backward induction (29) becomes,

$$\begin{aligned} {\mathcal {B}}_{k}= & {} \left\{ \begin{array}{llll} \{0\} &{} \text{ if } c^i (0;\Theta _{-i} )=c^i w_1 \le \nu _{k},\\ \{T\} &{} \text{ if } c^i (T;\Theta _{-i} ) >\nu _{k}, \\ \left\{ t:c^i(t;\Theta _{-i}) =\nu _{k} \right\} &{} \text{ else, } \end{array}\right. \end{aligned}$$

(C10)

because (a) $h_M(t)=c(t),$ so ${\mathcal {B}}_k$ is given by (C10) for $k=M$; (b) assume ${\mathcal {B}}_k$ is given by (C10) for $k=M\dots l+1$ and consider any $\theta _{l+1}\in {\mathcal {B}}_{l+1}$; (c) from part (i), ${\mathcal {B}}_l\subset [\theta _{l+1},T]$ and for any $t\in [\theta _{l+1},T]$, by Lemma 2$v_{l+1}(t)\le v_{l+1}(\theta _{l+1})=0$; hence $h_l(t)=c(t)$, and thus, by (29) ${\mathcal {B}}_l$ is also given by (C10). Proof of other parts follow from (C10). $\square$

Proof of Theorem 4:

From Theorem 3 and (C10), one can verify that only the tuples that satisfy (30) are NE, so second part is done. For the existence of NE, define the correspondence $g:[0,T]^{nM}\rightarrow [0,T]^{nM}$ where, $g(\cdot )=\{g^i_k(\cdot )\}_{i,k}$ with components defined as below,

$$\begin{aligned} g^i_k(\Theta )=\left\{ \arg \min _{0\le s\le T} \left( c^i(s;\Theta _{-i}) -\nu _{k}\right) ^2 \right\} , \text{ for } \text{ any } i,k, \end{aligned}$$

where $\Theta =(\Theta _{i},\Theta _{-i})\in [0,T]^{nM}$. It is easy to verify that the fixed point of the above correspondence (see [22]) is a NE. From the Maximum theorem (see [22]), $g^i_k(\cdot )$ is an upper semi continuous correspondence (uscc) for every i, k; so $g(\Theta )=\times _{i,k}g^i_k(\Theta ),$ is also uscc. Further, $g(\cdot )$ is non-empty, compact and convex for any $\Theta \in [0,T]^{nM}$, and $[0,T]^{nM}$ is compact and convex. Hence, from Kakutani’s fixed point theorem (see [22]), $g(\cdot )$ has a fixed point. $\square$

Proof of Proposition 1:

From Theorem 3, for any i given that $c^i w_1\le \nu _{k^i+1}$ for some $k^i$, any BR of agent i against any strategy of opponents will contain $\theta ^*_{k}=0$ for all $k \ge k^i+1$. Hence, the same is true for any NE. Further from (C8), as $\beta ^j \rightarrow 0$, the probabilities $Q^j(\cdot ,\cdot ,\cdot )$ converge to zero point-wise. This implies convergence of $P^i_m(\cdot ,\cdot )\rightarrow \mathcal{X}_{ \left\{ m=1\right\} }$ from (C6). Thus from (1), $c^i(T; \Theta _{-i}) \rightarrow c^i w_1.$ By continuity of $c^i(\cdot ;\Theta _{-i})$ function, there exists a ${\bar{\beta }}>0$ $\text{ such } \text{ that, } c^i (T; \Theta _{-i})>\nu _{k^i-1} \text{ for } \text{ all } \beta ^j\le {\bar{\beta }},\ k\ge k^i,$ with $k^i$ as in hypothesis. Thus, the proof can be completed using Theorem 3. $\square$