Abstract
Calibration of simulation models and hyperparameter optimisation of machine learning and deep learning methods are computationally demanding optimisation problems, for which many state-of-the-art optimisation methods are adopted and applied in various studies. However, their performances come to a test when the parameter optimisation problems exhibit high-dimensional spaces and expensive evaluation of models’ or methods’ settings. Population-based (evolutionary) methods work well for the former but not suitable for expensive evaluation functions. On the opposite, Bayesian optimisation eliminates the necessity of frequent simulations to find the global optima. However, the computational demand rises significantly as the number of parameters increases. Bayesian optimisation with random forests has overcome issues of its state-of-the-art counterparts. Still, due to the non-parametric output, it fails to utilise the capabilities of available acquisition functions. We propose a semi-parametric approach to overcome such limitations to random forests by identifying a mixture of parametric components in their outcomes. The proposed approach is evaluated empirically on four optimisation benchmark functions with varying dimensionality, confirming the improvement in guiding the search process. Finally, in terms of running time, it scales linearly with respect to the dimensionality of the search space.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
Code: https://tinyurl.com/2xtsaaut.
References
An, Z., Nott, D.J., Drovandi, C.: Robust Bayesian synthetic likelihood via a semi-parametric approach. Stat. Comput. 30(3), 543–557 (2020)
Aushev, A., Pesonen, H., Heinonen, M., Corander, J., Kaski, S.: Likelihood-free inference with deep Gaussian processes. arXiv preprint arXiv:2006.10571 (2020)
Bäck, T.: Evolutionary Algorithms in Theory and Practice: Evolution Strategies, Evolutionary Programming, Genetic Algorithms. Oxford University Press, USA (1996)
Bergstra, J., Bengio, Y.: Random search for hyper-parameter optimization. J. Mach. Learn. Res. 13(2), 281–305 (2012)
Blum, M., Nunes, M., Prangle, D., et al.: Comparative review of dimension reduction methods in approximate Bayesian computation. Stat. Sci. 28(2), 189–208 (2013)
Breiman, L.: Random forests. Mach. Learn. 45(1), 5–32 (2001)
Chen, B., Castro, R., Krause, A.: Joint optimization and variable selection of high-dimensional Gaussian processes. arXiv preprint arXiv:1206.6396 (2012)
Day, N.: Estimating the components of a mixture of normal components. Biometrika 56(3), 463–474 (1969)
Dempster, A.P., Laird, N.M., Rubin, D.B.: Maximum likelihood from incomplete data via the EM algorithm. J. R. Stat. Soc. Ser. B (Methodological) 39(1), 1–38 (1977)
Eggensperger, K., et al.: Towards an empirical foundation for assessing Bayesian optimization of hyperparameters. In: NIPS Workshop on BO in Theory and Practice (2013)
Falkner, S., Klein, A., Hutter, F.: BOHB: robust and efficient hyperparameter optimization at scale. In: International Conference on Machine Learning, pp. 1437–1446 (2018)
Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning. TSSCML, pp. 3–33. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-05318-5_1
Friedman, J.H., Hall, P.: On bagging and nonlinear estimation. J. Stat. Plann. Infer. 137(3), 669–683 (2007)
Gutmann, M.U., Corander, J.: Bayesian optimization for likelihood-free inference of simulator-based statistical models. J. Mach. Learn. Res. 17(1), 1–47 (2016)
Hansen, N.: The CMA evolution strategy: a tutorial. arXiv:1604.00772 (2016)
Hutter, F., Hoos, H.H., Leyton-Brown, K.: Sequential model-based optimization for general algorithm configuration. In: Coello, C.A.C. (ed.) LION 2011. LNCS, vol. 6683, pp. 507–523. Springer, Heidelberg (2011). https://doi.org/10.1007/978-3-642-25566-3_40
Järvenpää, M., Gutmann, M.U., Pleska, A., Vehtari, A., Marttinen, P., et al.: Efficient acquisition rules for model-based approximate Bayesian computation. Bayesian Anal. 14(2), 595–622 (2019)
Järvenpää, M., Gutmann, M.U., Vehtari, A., Marttinen, P., et al.: Gaussian process modelling in approximate Bayesian computation to estimate horizontal gene transfer in bacteria. Ann. Appl. Stat. 12(4), 2228–2251 (2018)
Kuzmanovski, V., Hollmén, J.: Composite surrogate for likelihood-free bayesian optimisation in high-dimensional settings of activity-based transportation models. In: Abreu, P.H., Rodrigues, P.P., Fernández, A., Gama, J. (eds.) IDA 2021. LNCS, vol. 12695, pp. 171–183. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-74251-5_14
Laguna, M., Marti, R.: Experimental testing of advanced scatter search designs for global optimization of multimodal functions. J. Glob. Optim. 33(2), 235–255 (2005)
Leclercq, F.: Bayesian optimization for likelihood-free cosmological inference. Phy. Rev. D 98(6) (2018)
Lintusaari, J., Gutmann, M., Dutta, R., Kaski, S., Corander, J.: Fundamentals and recent developments in approximate Bayesian computation. Syst. Biol. 66, e66–e82 (2017)
Locatelli, M. A Note on the Griewank Test Function. J. Glob. Optim. 25, 160–174 (2003). https://doi.org/10.1023/A:1021956306041
Meinshausen, N.: Quantile regression forests. JMLR 7, 983–999 (2006)
Mockus, J.: On Bayesian Methods for Seeking the Extremum. In: Marchuk, G.I. (eds.) Optimization Techniques IFIP Technical Conference. LNCS. Springer, Heidelberg (1975). https://doi.org/10.1007/978-3-662-38527-2_55
Oh, S., Seshadri, R., Azevedo, C., Ben-Akiva, M.E.: Demand calibration of multimodal microscopic traffic simulation using weighted discrete SPSA. Transp. Res. Rec. 2673(5), 503–514 (2019)
Petrik, O., Adnan, M., Basak, K., Ben-Akiva, M.: Uncertainty analysis of an activity-based microsimulation model for Singapore. Future. Gener. Comput. Sys. 110, 350–363 (2018)
Raynal, L., Marin, J., Pudlo, P., Ribatet, M., Robert, C., Estoup, A.: ABC random forests for Bayesian parameter inference. Bioinformatics 35(10), 1720–1728 (2019)
Schwefel, H.P.: Numerical Optimization of Computer Models. Wiley (1981)
Sha, D., Ozbay, K., Ding, Y.: Applying Bayesian optimization for calibration of transportation simulation models. Transp. Res. Rec. 2674, 036119812093625 (2020)
Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., De Freitas, N.: Taking the human out of the loop: a review of Bayesian optimization. Proc. IEEE 104(1), 148–175 (2015)
Sisson, S.A., Fan, Y., Beaumont, M.: Handbook of Approximate Bayesian Computation. CRC Press (2018)
Snoek, J., Rippel, O., Swersky, K., Kiros, R., Satish, N., Sundaram, N., Patwary, M., Prabhat, M., Adams, R.: Scalable Bayesian optimization using deep neural networks. In: International Conference on Machine Learning, pp. 2171–2180 (2015)
Springenberg, J.T., Klein, A., Falkner, S., Hutter, F.: Bayesian optimization with robust Bayesian neural networks. In: Proceedings of the 30th International Conference on Neural Information Processing Systems, pp. 4141–4149 (2016)
Titterington, D., Smith, A., Makov, U.: Statistical Analysis of Finite Mixture Distributions. Series in Probability and Mathematical Statistics. Wiley (1985)
Todorović, M., Gutmann, M., Corander, J., Rinke, P.: Bayesian inference of atomistic structure in functional materials. NPJ Comput. Mater. 5(1), 35 (2019)
Yu, T., Zhu, H.: Hyper-parameter optimization: a review of algorithms and applications. arXiv preprint arXiv:2003.05689 (2020)
Zhang, Y., Apley, D.W., Chen, W.: Bayesian optimization for materials design with mixed quantitative and qualitative variables. Sci. Rep. 10(1), 4924 (2020)
Acknowledgments
This work is supported by the European Commission through the H2020 project Finest Twins (grant No. 856602).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Kuzmanovski, V., Hollmén, J. (2022). Semi-parametric Approach to Random Forests for High-Dimensional Bayesian Optimisation. In: Pascal, P., Ienco, D. (eds) Discovery Science. DS 2022. Lecture Notes in Computer Science(), vol 13601. Springer, Cham. https://doi.org/10.1007/978-3-031-18840-4_30
Download citation
DOI: https://doi.org/10.1007/978-3-031-18840-4_30
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-18839-8
Online ISBN: 978-3-031-18840-4
eBook Packages: Computer ScienceComputer Science (R0)