Efficient Exploration by Novelty-Pursuit

Abstract

Efficient exploration is essential to reinforcement learning in tasks with huge state space and long planning horizon. Recent approaches to address this issue include the intrinsically motivated goal exploration processes (IMGEP) and the maximum state entropy exploration (MSEE). In this paper, we propose a goal-selection criterion in IMGEP based on the principle of MSEE, which results in the new exploration method novelty-pursuit. Novelty-pursuit performs the exploration in two stages: first, it selects a seldom visited state as the target for the goal-conditioned exploration policy to reach the boundary of the explored region; then, it takes random actions to explore the non-explored region. We demonstrate the effectiveness of the proposed method in environments from simple maze environments, MuJoCo tasks, to the long-horizon video game of SuperMarioBros. Experiment results show that the proposed method outperforms the state-of-the-art approaches that use curiosity-driven exploration.

Z. Li and X-H. Chen—The two authors contributed equally to this work.

A Appendix

1.1 A.1 Reward Shaping for Training Goal-Conditioned Policy

Reward shaping is invariant to the optimal policy under some conditions [27]. Here we verify that the reward shaping function introduced by our method doesn’t change the optimal behaviors for goal-conditioned policy. Lets’ consider the total shaping rewards during an episode of length T:

$$\begin{aligned}&\quad \sum _{t=1}^{T} -d(ag_t, g) + d(ag_{t+1}, g) \\&= -d(ag_1, g) + d(ag_2, g) - d(ag_2, g) + d(ag_3, g) \cdots \\&= -d(ag_1, g) + d(ag_{T+1}, g) \end{aligned}$$

For the optimal policy $\pi _g^*$, $d(ag_{T+1}, g) = 0$ while $d(ag_1, g)$ is a constant. Therefore, the optimal policy $\pi _g$ induced by the reward shaping is invariant to the one induced by the sparse reward function in Eq. 2.

Fig. 7.

Visualization for the true visitation counts and the corresponding exploration boundary.

1.2 A.2 Additional Results

In this part, we provide additional experiment results to better understand our method.

Empty Room. We visualize the true visitation counts and the corresponding exploration boundary in Fig. 7. Note the agent starts from the left top corner and the exit is on the most bottom right corner. The data used for visualization is collect by a random policy. Hence, the visitation counts are large on the left top part. We define the true exploration boundary as the top $10\%$ states with least visitation counts and the estimated exploration boundary given by our method are states with the largest prediction errors in the priority queue. From this figure, we can see that our method can make a good approximation to the true exploration boundary given by visitation counts.

SuperMarioBros. In Fig. 8, we make additional trajectory visualization on SuperMarioBros-1-1 and SuperMarioBros-1-2. Trajectories are plotted with the same number of samples (18M). We can observe that the vanilla method gets into the local optimum on SuperMarioBros-1-1 even though it has used the policy entropy regularization to encourage exploration. In addition, only our method can get the flag on SuperMarioBros-1-2.

Fig. 8.

Trajectory visualization on SuperMarioBros-1-1 and SuperMarioBros-1-2. For each figure, top row: vanilla ACER; middle row: ACER + exploration bonus; bottom row: novelty-pursuit (ours). The vanilla method gets stuck into the local optimum on SuperMarioBros-1-1. Only our method can get the flag on SuperMarioBros-1-2.

1.3 A.3 Environment Prepossessing

In this part, we present the used environment preprocessing.

Maze. Different from [9], we only use the image and coordination information as inputs. Also, we only consider four actions: turn left, turn right, move forward and move backward. The maximal episode length is 190 for Empty Room, and 500 for Four Rooms. Each time the agent receives a time penalty of $1/\mathrm {max\_ episode\_length}$ and receives a reward of +1 when it finds the exit.

FetchReach. We implement this environment based on FetchReach-v0 in Gym [5]. The maximal episode length is 50. The xyz coordinates of four spheres are (1.20, 0.90, 0.65), (1.10, 0.72, 0.45), (1.20, 0.50, 0.60), and (1.45, 0.50, 0.55). When sampling goals, we resample goals if the target position is outside of the table i.e., the valid x range: (1.0, 1.5), the valid y range is (0.45, 1.05), and the valid z range is (0.45, 0.65).

SuperMarioBros. We implement this environment based on [19] with OpenAI Gym wrappers. Prepossessing includes grey-scaling, observation downsampling, external reward clipping (except that 50 for getting flag), stacked frames of 4, and sticky actions with a probability of 0.25 [26]. The maximal episode length is 800. The environment restarts to the origin when the agent dies.

1.4 A.4 Network Architecture

We use the convolutional neural network (CNN) for Empty Room, Four Rooms, and video games of SuperMarioBros, and multi-layer perceptron (MLP) for FetchReach environment. Network architecture design and parameters are based on the default implementation in OpenAI baselines [11]. For each environment, RND uses a similar network architecture. However, the predictor network has additional MLP layers than the predictor network to strengthen its representation power [7].

1.5 A.5 Hyperparameters

Table 3 gives hyperparameters for ACER [43] on the maze and SuperMarioBros (the learning algorithm is RMSProp [41]. DDPG [23] used in Fetch Reach environments is based on the HER algorithm implemented in OpenAI baselines [11] expect that the actor learning rate is 0.0005. We run 4 parallel environments for DDPG and the size of the priority queue is also 100. As for the predictor network, the learning rate of the predictor network is 0.0005 and the optimization algorithm is Adam [20] for all experiments, and the batch size of training data is equal to the product of rollout length and the number of parallel environments.

The goal-conditioned exploration policy of our method is trained by combing the shaping rewards defined in Eq. 3 and environment rewards, which helps reduce the discrepancy with the exploitation policy. The weight for environment rewards is 1 for all environments except 2 for SuperMarioBros. For the bonus method used in Sect. 5, the weight $\beta$ to balance the exploration bonus is 0.1 for Empty Room and Four Rooms, 0.01 for FetchReach, 1.0 for SuperMarioBros-1-1 and SuperMarioBros-1-3, and 0.1 for SuperMarioBros-1-2. Following [6, 7] we also do a normalization for the exploration bonus by dividing them via a running estimate of the standard deviation of the sum of discounted exploration bonus. In addition, we find sometimes applying the imitation learning technique for the goal-conditioned policy can improve performance. We will examine this in detail in future works. Though we empirically find HER is useful in simple environments like the maze and the MuJoCo robotics tasks, we find it is less powerful than the technique of reward shaping on complicated tasks like SuperMarioBros. Hence, reported episode returns of learned policies are based on the technique of reward shaping on SuperMarioBros and HER for others.

For the exploitation policy, we periodically allow it to interact with the environment to mitigate the exploration error [16]. For all experiments, we split the half interactions for the exploitation method. For example, if the number of maximal samples is 200k, the exploration and the exploitation policy will use the same 100k interactions.

Table 3. Hyperparameters of our method based on ACER on the maze and SuperMarioBros environments.