Abstract
Hyperparameter optimization is one of the main pillars of machine learning algorithms. In this paper, we introduce Meta-Hyperband: a Hyperband based algorithm that improves the hyperparameter optimization by adding levels of exploitation. Unlike Hyperband method, which is a pure exploration bandit-based approach for hyperparameter optimization, our meta approach generates a trade-off between exploration and exploitation by combining the Hyperband method with meta-learning and Coarse-to-Fine modules. We analyze the performance of Meta-Hyperband on various datasets to tune the hyperparameters of CNN and SVM. The experiments indicate that in many cases Meta-Hyperband can discover hyperparameter configurations with higher quality than Hyperband, using similar amounts of resources. In particular, we discovered a CNN configuration for classifying CIFAR10 dataset which has a 3% higher performance than the configuration founded by Hyperband, and is also 0.3% more accurate than the best-reported configuration of the Bayesian optimization approach. Additionally, we release a publicly available pool of historically well-performed configurations on several datasets for CNN and SVM to ease the adoption of Meta-Hyperband.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
Our sources \(\rightarrow \) https://github.com/saminpayro/Meta_Hyperband_implementation.
- 2.
See “example layers” directory in http://code.google.com/p/cuda-convnet/.
- 3.
References
Bergstra, J., Bengio, Y.: Random search for hyper-parameter optimization. J. Mach. Learn. Res. 13, 281–305 (2012)
Bergstra, J., Rémi B., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Neural Information Processing Systems, pp. 2546–2554 (2011)
Charniak, E., Johnson, M.: Coarse-to-fine n-best parsing and maxent discriminative reranking. In: Annual Meeting on Association for Computational Linguistics, pp. 173–180. ACL 2005 (2005)
Deng, L., Li, X.: Machine learning paradigms for speech recognition: an overview. Trans. Audio Speech Lang. Process. 21(5), 1060–1089 (2013)
Feurer, M., Springenberg, J.T., Hutter, F.: Initializing bayesian hyperparameter optimization via meta-learning. In: AAAI Conference on Artificial Intelligence (2015)
Guo, G., Li, S.Z., Chan, K.: Face recognition by support vector machines. In: International Conference on Automatic Face and Gesture Recognition (2000)
Hutter, F., Hoos, H.H., Leyton-Brown, K.: Sequential model-based optimization for general algorithm configuration. In: Coello, C.A.C. (ed.) LION 2011. LNCS, vol. 6683, pp. 507–523. Springer, Heidelberg (2011). https://doi.org/10.1007/978-3-642-25566-3_40
Jamieson, P., Talwalkar, A.: Non-stochastic best arm identification and hyperparameter optimization. AISTATS (2015)
Krizhevsky, A.: Learning multiple layers of features from tiny images. Technical report, Department of Computer Science, University of Toronto (2009)
LeCun, Y., Bottou, L., Bengio, Y., Haffner, P.: Gradient-based learning applied to document recognition. Proc. IEEE 86(11), 2278–2324 (1998)
Li, L., Jamieson, K., DeSalvo, G., Rostamizadeh, A., Talwalkar, A.: Hyperband: a novel bandit-based approach to hyperparameter optimization. In: ICLR (2017)
Moshkelgosha, V., Behzadi-Khormouji, H., Yazdian-Dehkordi, M.: Coarse-to-fine parameter tuning for content-based object categorization. In: International Conference on Pattern Recognition and Image Analysis (IPRIA), pp. 160–165. IEEE (2017)
Netzer, Y., Wang, T., Coates, A., Bissacco, A., Wu, B., Ng, YA.: Reading digits in natural images with unsupervised feature learning. In: NIPS (2011)
Rejani, Y.I.A., Selvi, S.T.: Early detection of breast cancer using SVM classifier technique. CoRR abs/0912.2314 (2009)
Rolland, P., Scarlett, J., Bogunovic, I., Cevher, V.: High-dimensional bayesian optimization via additive models with overlapping groups. AISTATS (2018)
Sadeghi, A., Graux, D., Yazdi, H.S., Lehmann, J.: MDE: multi distance embeddings for link prediction in knowledge graphs. In: 24th European Conference on Artificial Intelligence (ECAI) (2020)
Snoek, J., Larochelle, H., Adams, R.: Practical bayesian optimization of machine learning algorithms. In: Neural Information Processing Systems (NIPS) (2012)
Vilalta, R., Drissi, Y.: A perspective view and survey of meta-learning. Artif. Intell. Rev. 18(2), 77–95 (2002)
Xiao, H., Rasul, K., Vollgraf, R.: Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747 (2017)
Acknowledgments
This study is supported by MLwin (Maschinelles Lernen mit Wissensgraphen, GA 01IS18050F of the German Federal Ministry of Education and Research), by the EU project Cleopatra (GA 812997) and by the Marie Skłodowska-Curie GA 801522 at the ADAPT SFI Research Centre (grant 13/RC/2106).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Payrosangari, S., Sadeghi, A., Graux, D., Lehmann, J. (2020). Meta-hyperband: Hyperparameter Optimization with Meta-learning and Coarse-to-Fine. In: Analide, C., Novais, P., Camacho, D., Yin, H. (eds) Intelligent Data Engineering and Automated Learning – IDEAL 2020. IDEAL 2020. Lecture Notes in Computer Science(), vol 12490. Springer, Cham. https://doi.org/10.1007/978-3-030-62365-4_32
Download citation
DOI: https://doi.org/10.1007/978-3-030-62365-4_32
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-62364-7
Online ISBN: 978-3-030-62365-4
eBook Packages: Computer ScienceComputer Science (R0)