Distributed Competitive Decision Making Using Multi-Armed Bandit Algorithms

Abstract

This paper tackles the problem of Opportunistic Spectrum Access (OSA) in the Cognitive Radio (CR). The main challenge of a Secondary User (SU) in OSA is to learn the availability of existing channels in order to select and access the one with the highest vacancy probability. To reach this goal, we propose a novel Multi-Armed Bandit (MAB) algorithm called $\epsilon$-UCB in order to enhance the spectrum learning of a SU and decrease the regret, i.e. the loss of reward by the selection of worst channels. We corroborate with simulations that the regret of the proposed algorithm has a logarithmic behavior. The last statement means that within a finite number of time slots, the SU can estimate the vacancy probability of targeted channels in order to select the best one for transmitting. Hereinafter, we extend $\epsilon$-UCB to consider multiple priority users, where a SU can selfishly estimate and access the channels according to his prior rank. The simulation results show the superiority of the proposed algorithms for a single or multi-user cases compared to the existing MAB algorithms.

Appendices

Appendix 1

In this Appendix, we investigate the upper bound of the term $\mathbb {A}= \epsilon _t \times Prob$ in e-UCB where Prob can be expressed as follows:

$$\begin{aligned} Prob \le P\big \{B_i(t-1, T_i(t-1)) \ge B_1(t-1, T_1(t-1)); T_i(t-1)\ge l \big \} \end{aligned}$$

The index of the i-th channel $B_i(t, T_i(t))$ is the sum of the exploration factor, $X_i(T_i(t))$, and the exploitation factor, $A_i(t, T_i(t))$:

$$\begin{aligned} B_i(t, T_i(t))= X_i(T_i(t))+ A_i(t, T_i(t)) \end{aligned}$$

(28)

Then, we obtain:

$$\begin{aligned}&Prob \le P\bigg \{X_1({T_1(t-1)}) + A_1(t-1, T_1(t-1)) \le \nonumber \\&\quad X_i({T_i(t-1)}) + A_i(t-1, T_i(t-1)) \text { and } T_i(t-1)\ge l \bigg \} \end{aligned}$$

(29)

By taking the minimum value of $X_1({T_1(t-1)}) + A_1(t-1, T_1(t-1))$ and the maximum value of $X_i({T_i(t-1)}) + A_i(t-1, T_i(t-1))$ at each time slot, we can upper bound Prob by the following equation:

$$\begin{aligned} Prob \le P\Bigg \{ \min _{0<S_1<t} \bigg [ X_1({S_1}) + A_1(t,S_1) \bigg ] \le \max _{l\le S_i<t} \bigg [ X_i({S_i}) + A_i(t,S_i) \bigg ] \Bigg \} \end{aligned}$$

(30)

where $S_i \ge l$ to fulfill the condition $T_i(t-1) \ge l$. Then, we obtain:

$$\begin{aligned} Prob \le \sum _{S_1=1}^{t-1} \sum _{S_i=l}^{t-1} P\Bigg \{X_1({S_1}) + A_1(t,S_1) < X_i(S_i)+A_i(t,S_i) \Bigg \} \end{aligned}$$

(31)

The above probability can be upper bounded by:

$$\begin{aligned}&Prob \le \sum _{S_1=1}^{t-1} \sum _{S_i=l}^{t-1} P\Big \{ X_1({S_1})+A_1({t,S_1}) \le \mu _1 \Big \} \nonumber \\&\qquad + P\Big \{\mu _1<\mu _i+2 A_i({t,S_i})\Big \} \nonumber \\&\qquad + P\Big \{ X_i(S_i)+ A_i({t,S_i}) \ge \mu _i+2A_i({t,S_i}) \Big \} \end{aligned}$$

(32)

Using the ceiling operator $\lceil \rceil$, let $l=\lceil \frac{4 \alpha \ln (n)}{\Delta _{i}^2}\rceil$, where $\Delta _{i} = \mu _1-\mu _i$ and $S_i\ge l$, then the inequality $\mu _1<\mu _i+2 A_i({t,S_i})$ in Eq. (32) becomes false, in fact:

$$\begin{aligned} \mu _1-\mu _i-2 A_i({t,S_i})&= \mu _1-\mu _i-2\sqrt{\frac{\alpha \ln (t)}{S_i} }\\&\ge \mu _1-\mu _i-2\sqrt{\frac{\alpha \ln (n)}{l} } \\&\ge \mu _1-\mu _i - \Delta _{i} = 0 \end{aligned}$$

Based on Eq. (32), we obtain:

$$\begin{aligned} Prob \le \sum _{S_1=1}^{t-1} \sum _{S_i=l}^{t-1} P\bigg \{X_1({S_1})\le \mu _1 - A_1( t,S_1) \bigg \} + P\bigg \{X_i(S_i)\ge \mu _i + A_i(t,S_i) \bigg \} \end{aligned}$$

(33)

Using Chernoff–Hoeffding bound⁴ [24], we can prove that:

$$\begin{aligned} P\Big \{X_1({S_1}) \le \mu _1-A_1({t,S_1})\Big \}&\le \exp ^{ \frac{-2}{S_1}\Big [S_1 \sqrt{\frac{\alpha \ln (t)}{S_1}} \Big ]^2 } \nonumber \\&= t^{-2\alpha } \end{aligned}$$

(34)

$$\begin{aligned} P\Big \{X_i(S_i)\ge \mu _i+A_i({t,S_i})\Big \}&\le \exp ^{ \frac{-2}{S_i}\Big [S_i \sqrt{\frac{\alpha \ln (t)}{S_i}} \Big ]^2 } \nonumber \\&= t^{-2\alpha } \end{aligned}$$

(35)

The two inequations above and inequation (33) lead us to:

$$\begin{aligned} Prob \le \sum _{S_1=1}^{t-1} \sum _{S_i=l}^{t-1} 2 t^{-2 \alpha } \le 2 t^{-2\alpha +2} \end{aligned}$$

(36)

Finally, we obtain:

$$\begin{aligned} \mathbb {A} \le \frac{H}{t} \times 2 t^{-2\alpha +2} = 2H \times t^{-2\alpha +1} \end{aligned}$$

(37)

Appendix 2

This appendix stands for finding an upper bound of Z that contributes to finding an upper bound of e-UCB:

$$\begin{aligned} Z= p\big \{X_1(T_1(t-1)) \le a ; T_i(t-1)\ge l\ \big \} \end{aligned}$$

(38)

where a is a constant number that can be chosen as follows: $a= \frac{\mu _1 + \mu _i}{2}= \mu _1-\frac{\Delta _i}{2}= \mu _i+\frac{\Delta _i}{2}$, and $\Delta _i=\mu _1-\mu _i$ . After the learning period where $T_i(t-1)\ge l$, we have: $T_1(t-1)>>T_i(t-1)$. Then Z can be upper bounded by:

$$\begin{aligned} Z \le&p\big \{X_1(T_1(t-1)) \le a ; T_1(t-1)\ge l\ \big \}\nonumber \\ \le&\sum _{z=l}^{n} p\big \{X_1(T_1(t-1)) \le \mu _1-\frac{\Delta _i}{2}; T_1(t-1) = z \big \} \nonumber \\ \le&\sum _{z=l}^{n} p\big \{X_1(z) \le \mu _1-\frac{\Delta _i}{2}\big \} \end{aligned}$$

(39)

Using the Chernoff–Hoeffding [24], we can upper bound the above equation as follows:

$$\begin{aligned} Z \le \sum _{z=l}^{n} \exp ^{- \frac{2 \Delta _i^2 z^2}{4z}} \le n \exp ^{\frac{- l\Delta _i^2}{2}} \end{aligned}$$

(40)

According to the proof provided in Appendix 1, we have $l=\lceil \frac{4 \alpha \ln (n)}{\Delta _{i}^2}\rceil$ where $\alpha =2$. So, we obtain:

$$\begin{aligned} Z \le n \exp ^ {-4\ln n} = \frac{1}{n^3} \end{aligned}$$

(41)