Which Hype for My New Task? Hints and Random Search for Echo State Networks Hyperparameters

Abstract

In learning systems, hyperparameters are parameters that are not learned but need to be set a priori. In Reservoir Computing, there are several parameters that needs to be set a priori depending on the task. Newcomers to Reservoir Computing cannot have a good intuition on which hyperparameters to tune and how to tune them. For instance, beginners often explore the reservoir sparsity, but in practice this parameter is not of high influence on performance for ESNs. Most importantly, many authors keep doing suboptimal hyperparameter searches: using grid search as a tool to explore more than two hyperparameters, while restraining the spectral radius to be below unity. In this short paper, we give some suggestions, intuitions, and give a general method to find robust hyperparameters while understanding their influence on performance. We also provide a graphical interface (included in ReservoirPy) in order to make this hyperparameter search more intuitive. Finally, we discuss some potential refinements of the proposed method.

5 Supplementary Material

1.1 5.1 Implementation details

We used the ReservoirPy library [5], which has been interfaced with hyperopt [4], to make the following experiments and figures in order to dive into the exploration of HPs. We illustrated our proposed method with two different tasks of chaotic time series prediction. These tasks consist in predicting the value of the series at time step $t+1$ given its value at time step t. To asses the performance of our models on these tasks, we performed cross validation. Each fold is composed of a train and a validation data set. The folds are defined as continuous slices of the original series, with an overlap in time, e.g. the validation set of the first fold would be the training set of the second, and the two sets would be two adjacent sequences of data in time in the series. For the last fold of the time series, the train set is defined as the last available slice of data, while the train set of the first fold is used as validation set.

For all tasks, we used a 3-fold cross validation measure on a time series composed of 6000 time steps, i.e. each fold is composed of 4000 time steps, with a train set and a validation set of 2000 time steps each. We used two metrics to perform this measure: the Normalized Root Mean Squared Error, defined in Eq. 1, and the $R^2$ correlation coefficient, defined in Eq. 2:

$$\begin{aligned} \mathrm {NRMSE}(y, \hat{y}) = \frac{ \sqrt{ \frac{1}{N}\sum _{t=0}^{N-1} (y_t - \hat{y}_t)^2 }}{\max y - \min y} \end{aligned}$$

(1)

$$\begin{aligned} R^2(y, \hat{y}) = 1 - \frac{\sum _{t=0}^{N-1}(y_t - \hat{y}_t)^2}{\sum _{t=0}^{N-1}(y_t - \bar{y})^2} \end{aligned}$$

(2)

where y is a time series defined over N time steps, $\hat{y}$ is the estimated time series predicted by the model, and $\bar{y}$ is the average value of the time series y over time. NRMSE was used as an error measure, which we expect to reach a value near 0, while $R^2$ was used as a score, which we expect to reach 1, its maximum possible value. All measures were made by averaging this two metrics across all folds, with 5 different initializations of the models for each fold (Figs. 5 and 6).

Fig. 5.

Restricted search on the IS parameter, for the Mackey-Glass task. The fixed value of 1 defined at the beginning of the search for IS might not be the optimal value given all the other parameters chosen. In the case of the Mackey-Glass task, we can clearly see in the top-left and bottom-left plots that better results are achieved with lower values, with the top 10% trials distribution being placed around a median of 0.3.

Fig. 6.

Large search dependence plot for Lorenz task, with 125 trials. Diagonal of the plot matrix displays interactions between all parameters explored and the error value, also referred to as loss value. Top 10% trials in terms of score (here, $R^{2}$) are represented using colors from yellow (top 10) to red (top 1) in all plots. Other plots on the matrix display the interactions of all possible couple of parameters. In these plots, the value of the error is represented using shades of purple and blue, on a logarithmic scale, while score is represented using different circle sizes. Circle size is normalized regarding to the score values. Because $R^{2}$ can take values between $-\infty$ and 1, the smallest dots represents negative values. Finally, the bottom row of plots display the parameter distribution of the top 10% of trials, in terms of score, and for each parameter.