Lucid dreaming for experience replay: refreshing past states with the current policy

Abstract

Experience replay (ER) improves the data efficiency of off-policy reinforcement learning (RL) algorithms by allowing an agent to store and reuse its past experiences in a replay buffer. While many techniques have been proposed to enhance ER by biasing how experiences are sampled from the buffer, thus far they have not considered strategies for refreshing experiences inside the buffer. In this work, we introduce L uc i d D reaming for E xperience R eplay (LiDER), a conceptually new framework that allows replay experiences to be refreshed by leveraging the agent’s current policy. LiDER consists of three steps: First, LiDER moves an agent back to a past state. Second, from that state, LiDER then lets the agent execute a sequence of actions by following its current policy—as if the agent were “dreaming” about the past and can try out different behaviors to encounter new experiences in the dream. Third, LiDER stores and reuses the new experience if it turned out better than what the agent previously experienced, i.e., to refresh its memories. LiDER is designed to be easily incorporated into off-policy, multi-worker RL algorithms that use ER; we present in this work a case study of applying LiDER to an actor–critic-based algorithm. Results show LiDER consistently improves performance over the baseline in six Atari 2600 games. Our open-source implementation of LiDER and the data used to generate all plots in this work are available at https://github.com/duyunshu/lucid-dreaming-for-exp-replay.

Appendices

Appendices for “lucid dreaming for experience replay: refreshing past states with the current policy”

We provide further details of our work in the following six appendices:

• “Appendix 1” contains the implementation details of LiDER, including neural network architecture, hyperparameters, and computation resources used for all experiments.

• “Appendix 2” presents the pseudocode for the A3C and SIL workers. Both follow the original work of Mnih et al. [28] and Oh et al. [32], respectively, we add them here for completeness.

• “Appendix 3” provides detailed statistics of the one-tailed independent-samples t tests: 1) A3CTBSIL compared to LiDER, 2) A3CTBSIL compared to the three ablation studies of LiDER, 3) A3CTBSIL compared to the two extensions of LiDER, 4) LiDER compared to the three ablation studies of LiDER, and 5) LiDER compared to the two extensions of LiDER.

• “Appendix 4” discusses the differences between the A3CTBSIL algorithm in Jr et al. [6] and the original SIL algorithm in Oh et al. [32] (as mentioned in Sect. 4.1).

• “Appendix 5” presents the performance of the trained agents (TA) used in LiDER-TA.

• “Appendix 6” details the pre-training process for obtaining the BC models used in LiDER-BC, including the statistics of the demonstration collected by de la Cruz et al. [5], the network architecture, the hyperparameters used for pre-training, and the performance of the trained BC models.

Appendix 1: Implementation details

We use the same neural network architecture as in the original A3C algorithm [28] for all A3C, SIL, and refresher workers (the blue, orange, and green components in Fig. 1, respectively). The network consists of three convolutional layers, one fully connected layer, followed by two branches of a fully connected layer: a policy function output layer and a value function output layer. Atari images are converted to grayscale and resized to 88$\times$88 with 4 images stacked as the input.

We run each experiment for eight trials due to computation limitations. Each experiment uses one GPU (Tesla K80 or TITAN V), five CPU cores, and 40 GB of memory (each LiDER-OneBuffer experiment uses 64 GB of memory since the buffer size was doubled). The refresher worker runs on GPU to generate data as quickly as possible; the A3C and SIL workers run distributively on CPU cores. In all games, the wall-clock time is roughly 0.8 to 1 million steps per hour and around 50 to 60 hours to complete one trial of 50 million steps.

The baseline A3CTBSIL is trained with 17 parallel workers; 16 A3C workers and 1 SIL worker. The RMSProp optimizer is used with a learning rate = 0.0007. We use $t_{\mathrm{max}}=20$ for n-step bootstrap $Q^{(n)}$ ($n \le t_{\mathrm{max}}$). The SIL worker performs $M=4$ SIL policy updates (Equation (2)) per step t with minibatch size 32 (i.e., 32$\times$4=128 total samples per step). Buffer ${{\,\mathrm{\mathcal {D}}\,}}$ is of size $10^5$. The SIL loss weight $\beta ^{\mathrm{sil}}=0.5$.

LiDER is also trained with 17 parallel workers: 15 A3C workers, 1 SIL worker, and 1 refresher worker—we keep the total number of workers in A3CTBSIL and LiDER the same to ensure a fair performance comparison. The SIL worker in LiDER also uses a minibatch size of 32; samples are taken from buffer ${{\,\mathrm{\mathcal {D}}\,}}$ and ${{\,\mathrm{\mathcal {R}}\,}}$ as described in Sect. 3. All other parameters are identical to that of A3CTBSIL. We summarize the details of the network architecture and experiment parameters in Table 2.

Table 2 Hyperparameters for all experiments. We train each game for 50 million steps with a frame skip of 4, i.e., 200 million game frames were consumed for training

Appendix 2: Pseudocode for the A3C and SIL workers

Appendix 3: One-tailed independent-samples t tests

We conducted one-tailed independent-samples t tests (equal variances not assumed) in all games to compare the differences in the mean episodic reward among all methods in this paper. For each game, we restored the best model checkpoint from each trial (eight trials per method) and executed the model in the game following a deterministic policy for 100 episodes (an episode ends when the agent loses all its lives) and recorded the reward per episode. This gives us 800 data points for each method in each game. We use a significance level $\alpha =0.001$ for all tests.

First, we check the statistical significance of the baseline A3CTBSIL compared to LiDER (Sect. 4.1), the main framework proposed in this paper. We report the detailed statistics in Table 3. Results show that the mean episodic reward of LiDER is significantly higher than A3CTBSIL ($p \ll 0.001$) in all games.

Table 3 One-tailed independent-samples t test for the differences of the mean episodic reward between A3CTBSIL and LiDER. Equal variances are not assumed

Second, we compare A3CTBSIL to the three ablation studies, LiDER-AddAll, LiDER-OneBuffer, and LiDER-SampleR (Sect. 5). Table 4 shows that all ablations were helpful in Freeway and Montezuma’s Revenge, in which the mean episodic rewards of the ablations are significantly higher than the baseline ($p \ll 0.001$). LiDER-AddAll also performed significantly better than A3CTBSIL in all games ($p \ll 0.001$). LiDER-OneBuffer outperformed A3CTBSIL in Freeway and Montezuma’s Revenge ($p \ll 0.001$), but it performed worse than the other four games ($p \ll 0.001$). LiDER-SampleR outperformed A3CTBSIL in Ms. Pac-Man, Freeway, and Montezuma’s Revenge ($p \ll 0.001$), but under-performed A3CTBSIL in Gopher, NameThisGame, and Alien ($p \ll 0.001$).

Table 4 One-tailed independent-samples t test for the differences of the mean episodic reward between A3CTBSIL and LiDER-AddALL, between A3CTBSIL and LiDER-OneBuffer, and between A3CTBSIL and LiDER-SampleR. Equal variances are not assumed

Third, we compare A3CTBSIL to the two extensions, LiDER-BC and LiDER-TA (Sect. 6). Table 5 shows that the two extensions outperformed the baseline significantly in all games ($p \ll 0.001$).

Table 5 One-tailed independent-samples t test for the differences of the mean episodic reward between A3CTBSIL and LiDER-BC, and between A3CTBSIL and LiDER-TA. Equal variances are not assumed

Fourth, we check the statistical significance of LiDER compared to the three ablation studies, LiDER-AddAll, LiDER-OneBuffer, and LiDER-SampleR (Sect. 5). Results in Table 6 show that most of the ablations significantly under-performed LiDER ($p \ll 0.001$) in terms of the mean episodic reward. Except for Gopher and NameThisGame, in which LiDER-AddAll performs at the same level as LiDER ($p > 0.001$).

Table 6 One-tailed independent-samples t test for the differences of the mean episodic reward between LiDER and LiDER-AddAll, between LiDER and LiDER-OneBuffer, and between LiDER and LiDER-SampleR. Equal variances are not assumed. Methods in bold are not significant at level $\alpha =0.001$

Lastly, we compare LiDER to the two extensions, LiDER-TA and LiDER-BC (Sect. 6). Results in Table 7 show that LiDER-TA always outperforms LiDER ($p \ll 0.001$). LiDER-BC outperformed LiDER in Gopher, Alien, Ms. Pac-Man, and Montezuma’s Revenge. In Freeway, LiDER-BC performs the same as LiDER ($p > 0.001$), while in NameThisGame LiDER-BC performed worse than LiDER ($p \ll 0.001$).

Table 7 One-tailed independent-samples t test for the differences of the mean episodic reward between LiDER and LiDER-TA, and between LiDER and LiDER-BC. Equal variances are not assumed. Methods in bold are not significant at level $\alpha =0.001$

Appendix 4: Differences between A3CTBSIL and SIL

There is a performance difference in Montezuma’s Revenge between the A3CTBSIL algorithm (our previous work in de la Cruz Jr et al. [6], which is used as the baseline method in this article) and the original SIL algorithm (by Oh et al. [32]). The A3CTBSIL agent fails to achieve any reward while the SIL agent can achieve a score of 1100 (Table 5 in [32]).

We hypothesize that the difference is due to the different number of SIL updates (Equation (2)) that can be performed in A3CTBSIL and SIL; lower numbers of SIL updates would decrease the performance. In particular, Oh et al. [32] proposed to add the “Perform self-imitation learning” step in each A3C worker (Algorithm 1 of Oh et al. [32]). That is, when running with 16 A3C workers, the SIL agent is actually using 16 SIL workers to update the policy. However, A3CTBSIL only has one SIL worker, which means A3CTBSIL performs strictly fewer SIL updates compared to that of the original SIL algorithm, and thus resulting in lower performance.

We empirically validate the above hypothesis by conducting an experiment in the game of Ms. Pac-Man by modifying the A3CTBSIL algorithm from our previous work [6]. Instead of performing a SIL update whenever the SIL worker can, we force the SIL worker to only perform an update at even global steps; this setting reduces the total number of SIL updates by half. We denote this experiment as A3CTBSIL-ReduceSIL.

Figure 8 shows that A3CTBSIL-ReduceSIL under-performed A3CTBSIL, which provides preliminary evidence that the number of SIL updates is positively correlated to performance. More experiments will be performed in future work to further validate this correlation.

Fig. 8

A3CTBSIL-ReduceSIL compared to A3CTBSIL in the game of Ms. Pac-Man. The x-axis is the total number of environmental steps. The y-axis is the average testing score over five trials. We ran A3CTBSIL-ReduceSIL for five trials due to limited computing resources; we plot the first five trials out of eight for A3CTBSIL for a fair comparison to the number of trials in A3CTBSIL-ReduceSIL. Shaded regions show the standard deviation

Appendix 5: The performance of trained agents used in LiDER-TA

Section 6 shows that LiDER can leverage knowledge from a trained agent (TA). While the TA could come from any source, we use the best checkpoint of a fully trained LiDER agent. Table 8 shows the average performance of the TA used in each game. The score is estimated by executing the TA greedily in the game for 50 episodes. An episode ends when the agent loses all its lives.

Table 8 The performance of trained agents used in LiDER-TA, shown as the purple dotted line in Fig. 7. The score is estimated by executing the TA greedily in the game for 50 episodes

Appendix 6: Pre-training the behavior cloning model for LiDER-BC

In Sect. 6, we demonstrated that a BC model can be incorporated into LiDER to improve learning. The BC model is pre-trained using a publicly available human demonstration dataset. Dataset statistics are shown in Table 9.

Table 9 Demonstration size and quality, collected in de la Cruz et al. [5]. All games are limited to 20 min of demonstration time per episode

The BC model uses the same network architecture as the A3C algorithm [28] and pre-training a BC model for A3C requires a few more steps than just using supervised learning as to how it is normally done in standard imitation learning (e.g., Ross and Bagnell [35]). A3C has two output layers: a policy output layer and a value output layer. The policy output is what we usually train a supervised classifier for. However, the value output layer is usually initialized randomly without being pre-trained. Our previous work [6] observed this inconsistency and leveraged demonstration data to also pre-train the value output layer. In particular, since the demonstration data contains the true return ${\mathrm{G}}$, we can obtain a value loss that is almost identical to A3C’s value loss $L^{a3c}_{\mathrm{value}}$: instead of using the n-step bootstrap value $Q^{(n)}$ to compute the advantage, the true return ${\mathrm{G}}$ is used.

Inspired by the supervised autoencoder (SAE) framework [22], our previous work [6] also blended in an unsupervised loss for pre-training. In SAE, an image reconstruction loss is incorporated with the supervised loss to help extract better feature representations and achieve better performance. A BC model pre-trained jointly with supervised, value, and unsupervised losses can lead to better performance after fine-tuning with RL, compared to pre-training with the supervised loss only.

We copy this approach by jointly pre-training the BC model for 50,000 steps with a minibatch of size 32. Adam optimizer is used with a learning rate = 0.0005. After training, we perform testing for 50 episodes by executing the model greedily in the game and record the average episodic reward (an episode ends when the agent loses all its lives). For each set of demonstration data, we train five models and use the one with the highest average episodic reward as the BC model in LiDER-BC. The performance of the trained BC models is present in Table 10. All parameters are based on those from our previous work [6] and we summarize them in Table 11.

Table 10 The performance of behavior cloning models used in LiDER-BC, shown as the black dashed line in Fig. 7. The score is estimated by executing the BC greedily in the game for 50 episodes

Table 11 Hyperparameters for pre-training the behavior cloning (BC) model used in LiDER-BC