Skip to main content
SpringerLink
Log in

HybridTuner: Tuning with Hybrid Derivative-Free Optimization Initialization Strategies

  • Conference paper
  • First Online:
Learning and Intelligent Optimization (LION 2021)

Part of the book series: Lecture Notes in Computer Science ((LNTCS,volume 12931))

Included in the following conference series:

Abstract

To utilize the full potential of advanced computer architectures, algorithms often need to be tuned to the architecture being used. We propose two hybrid derivative-free optimization (DFO) methods to maximize the performance of an algorithm after evaluating a small number of possible algorithmic configurations. Our autotuner (a) reduces the execution time of dense matrix multiplication by a factor of 1.4\(\times \) compared to state-of-the-art autotuners, (b) identifies high-quality tuning parameters within only \(5\%\) of the computational effort required by other autotuners and (c) can be applied to any computer architecture.

This work was conducted as part of the Institute for the Design of Advanced Energy Systems (IDAES) with funding from the Office of Fossil Energy, Cross-Cutting Research, U.S. Department of Energy. This work used the Extreme Science and Engineering Discovery Environment (XSEDE), which is supported by National Science Foundation grant number ACI-1548562. Specifically, it used the Bridges system, which is supported by NSF award number ACI-1445606, at the Pittsburgh Supercomputing Center (PSC). We also gratefully acknowledge the support of the NVIDIA Corporation with the donation of the NVIDIA Tesla K40 GPU used for this research.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 69.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 89.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Adams, B.M., et al.: DAKOTA, A Multilevel Parallel Object-Oriented Framework for Design Optimization, Parameter Estimation, Uncertainty Quantification, and Sensitivity Analysis: Version 6.5 User’s Manual. Sandia National Laboratories, Albuquerque/Livermore (2016). https://dakota.sandia.gov/

  2. Ansel, J., et al.: PetaBricks: a language and compiler for algorithmic choice. In: Proceedings of the 30th ACM SIGPLAN Conference on Programming Language Design and Implementation, pp. 38–49. Association for Computing Machinery, New York (2009)

    Google Scholar 

  3. Ansel, J., et al.: OpenTuner: an extensible framework for program autotuning. In: Proceedings of the 23rd International Conference on Parallel Architectures and Compilation, pp. 303–316. Association for Computing Machinery, New York (2014)

    Google Scholar 

  4. Ashouri, A., Mariani, G., Palermo, G., Park, E., Cavazos, J., Silvano, C.: COBAYN: compiler autotuning framework using Bayesian networks. ACM Trans. Archit. Code Optim. (TACO) 13, 1–26 (2016)

    Article  Google Scholar 

  5. Audet, C., Dang, C.-K., Orban, D.: Algorithmic parameter optimization of the DFO method with the OPAL framework. In: Suda, R., Naono, K., Teranishi, K., Cavazos, J. (eds.) Software Automatic Tuning, pp. 255–274. Springer, New York (2011). https://doi.org/10.1007/978-1-4419-6935-4_15

    Chapter  Google Scholar 

  6. Audet, C., Orban, D.: Finding optimal algorithmic parameters using derivative-free optimization. Soc. Ind. Appl. Math. 17, 642–664 (2006)

    MathSciNet  MATH  Google Scholar 

  7. Balandat, M., et al.: BoTorch: programmable Bayesian optimization in PyTorch, pp. 1–20. arXiv preprint arXiv:1910.06403 (2019)

  8. Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Shawe-Taylor, J., Zemel, R.S., Bartlett, P.L., Pereira, F., Weinberger, K.Q. (eds.) Proceedings of the 24th International Conference on Neural Information Processing Systems, pp. 2546–2554. Curran Associates Inc., Red Hook (2011)

    Google Scholar 

  9. Birattari, M., Yuan, Z., Balaprakash, P., Stützle, T.: F-Race and iterated F-Race: an overview. In: Bartz-Beielstein, T., Chiarandini, M., Paquete, L., Preuss, M. (eds.) Experimental Methods for the Analysis of Optimization Algorithms, pp. 311–336. Springer, Heidelberg (2010). https://doi.org/10.1007/978-3-642-02538-9_13

    Chapter  Google Scholar 

  10. Bruel, P., Gonzalez, M., Goldman, A.: Autotuning GPU compiler parameter using OpenTuner. In: XXII Symposium of Systems of High Performance Computing, Bangalore, India, pp. 1–12. IEEE (2015)

    Google Scholar 

  11. Carter, R., Gablonsky, J., Patrick, A., Kelley, C., Eslinger, O.: Algorithms for noisy problems in gas transmission pipeline optimization. Optim. Eng. 2, 139–157 (2001). https://doi.org/10.1023/A:1013123110266

    Article  MathSciNet  MATH  Google Scholar 

  12. Custódio, A.L., Vicente, L.N.: SID-PSM: a pattern search method guided by simplex derivatives for use in derivative-free optimization. Departamento de Matemática, Universidade de Coimbra, Coimbra, Portugal (2008)

    Google Scholar 

  13. Davidson, A., Owens, J.: Toward techniques for auto-tuning GPU algorithms. In: Jónasson, K. (ed.) PARA 2010. LNCS, vol. 7134, pp. 110–119. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-28145-7_11

    Chapter  Google Scholar 

  14. Fan, S.S., Zahara, E.: A hybrid simplex search and particle swarm optimization for unconstrained optimization. Eur. J. Oper. Res. 181, 527–548 (2007)

    Article  MathSciNet  Google Scholar 

  15. Fialho, A., Da Costa, L., Schoenauer, M., Sebag, M.: Analyzing bandit-based adaptive operator selection mechanisms. Ann. Math. Artif. Intell. 60, 25–64 (2010). https://doi.org/10.1007/s10472-010-9213-y

    Article  MathSciNet  MATH  Google Scholar 

  16. Gray, G.A., Kolda, T.G.: Algorithm 856: APPSPACK 4.0: parallel pattern search for derivative-free optimization. ACM Trans. Math. Softw. 32, 485–507 (2006)

    Article  MathSciNet  Google Scholar 

  17. Griffin, J.D., Kolda, T.G.: Asynchronous parallel hybrid optimization combining DIRECT and GSS. Optim. Methods Softw. 25, 797–817 (2010)

    Article  MathSciNet  Google Scholar 

  18. Hemker, T., Werner, C.: DIRECT using local search on surrogates. Pac. J. Optim. 7, 443–466 (2011)

    MathSciNet  MATH  Google Scholar 

  19. Holmström, K., Göran, A.O., Edvall, M.M.: User’s Guide for TOMLAB 7. Tomlab Optimization. http://tomopt.com

  20. Hutter, F., Hoos, H.H., Leyton-Brown, K.: Sequential model-based optimization for general algorithm configuration. In: Coello, C.A.C. (ed.) LION 2011. LNCS, vol. 6683, pp. 507–523. Springer, Heidelberg (2011). https://doi.org/10.1007/978-3-642-25566-3_40

    Chapter  Google Scholar 

  21. Hutter, F., Hoos, H.H., Leyton-Brown, K., Stützle, T.: ParamILS: an antomatic algorithm configuration framework. J. Artif. Intell. Res. 36, 267–306 (2009)

    Article  Google Scholar 

  22. Huyer, W., Neumaier, A.: SNOBFIT-stable noisy optimization by branch and fit. ACM Trans. Math. Softw. 35, 1–25 (2008)

    Article  MathSciNet  Google Scholar 

  23. Jones, D.R.: The DIRECT global optimization algorithm. In: Floudas, C.A., Pardalos, P.M. (eds.) Encyclopedia of Optimization, vol. 1, pp. 431–440. Kluwer Academic Publishers, Boston (2001)

    Chapter  Google Scholar 

  24. Kennedy, J., Eberhart, R.: Particle swarm optimization. In: Proceedings of the IEEE International Conference on Neural Networks, Piscataway, NJ, USA, pp. 1942–1948 (1995)

    Google Scholar 

  25. Li, Y., Dongarra, J., Tomov, S.: A note on auto-tuning GEMM for GPUs. In: Allen, G., Nabrzyski, J., Seidel, E., van Albada, G.D., Dongarra, J., Sloot, P.M.A. (eds.) ICCS 2009. LNCS, vol. 5544, pp. 884–892. Springer, Heidelberg (2009). https://doi.org/10.1007/978-3-642-01970-8_89

    Chapter  Google Scholar 

  26. López-Ibáñez, M., Dubois-Lacoste, J., Cáceres, L., Birattari, M., Stützle, T.: The irace package: iterated racing for automatic algorithm configuration. Oper. Res. Perspect. 3, 43–58 (2016)

    MathSciNet  Google Scholar 

  27. Loshchilov, I., Hutter, F.: CMA-ES for hyperparameter optimization of deep neural networks, pp. 1–15. arXiv preprint arXiv:1604.07269 (2016)

  28. Metropolis, N., Rosenbluth, A.W., Rosenbluth, M.N., Teller, A.H., Teller, E.: Equation of state calculations by fast computing machines. J. Chem. Phys. 21, 1087–1092 (1953)

    Article  Google Scholar 

  29. Nath, R., Tomov, S., Dongarra, J.: An improved MAGMA GEMM for Fermi graphics processing units. Int. J. High Perform. Comput. Appl. 24, 511–515 (2010)

    Article  Google Scholar 

  30. Nelder, J.A., Mead, R.: A simplex method for function minimization. Comput. J. 7, 308–313 (1965)

    Article  MathSciNet  Google Scholar 

  31. Nystrom, N., Levine, M., Roskies, R., Scott, J.: Bridges: a uniquely flexible HPC resource for new communities and data analytics. In: Proceedings of the 2015 XSEDE Conference: Scientific Advancements Enabled by Enhanced Cyberinfrastructure, pp. 1–8. Association for Computing Machinery, New York (2015)

    Google Scholar 

  32. Pacula, M., Ansel, J., Amarasinghe, S., O’Reilly, U.-M., et al.: Hyperparameter tuning in bandit-based adaptive operator selection. In: Di Chio, C. (ed.) EvoApplications 2012. LNCS, vol. 7248, pp. 73–82. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-29178-4_8

    Chapter  Google Scholar 

  33. Plantenga, T.D.: HOPSPACK 2.0 user manual. Technical report SAND2009-6265, Sandia National Laboratories, Albuquerque, NM and Livermore, CA (2009). https://software.sandia.gov/trac/hopspack/

  34. Powell, M.J.D.: UOBYQA: unconstrained optimization BY quadratic approximation. Math. Program. 92, 555–582 (2002). https://doi.org/10.1007/s101070100290

    Article  MathSciNet  MATH  Google Scholar 

  35. Rios, L.M., Sahinidis, N.V.: Derivative-free optimization: a review of algorithms and comparison of software implementations. J. Glob. Optim. 56, 1247–1293 (2013). https://doi.org/10.1007/s10898-012-9951-y

    Article  MathSciNet  MATH  Google Scholar 

  36. Sauk, B., Ploskas, N., Sahinidis, N.V.: GPU parameter tuning for tall and skinny dense linear least squares problems. Optim. Methods Softw. 35, 638–660 (2020)

    Article  MathSciNet  Google Scholar 

  37. Snoek, J., Larochelle, H., Adams, R.P.: Practical Bayesian optimization of machine learning algorithms. In: Pereira, F., Burges, C.J.C., Bottou, L., Weinberger, K.Q. (eds.) Proceedings of the 25th International Conference on Neural Information Processing Systems, pp. 2951–2959. Curran Associates Inc., Red Hook (2012)

    Google Scholar 

  38. Tan, G., Li, L., Triechle, S., Phillips, E., Bao, Y., Sun, N.: Fast implementation of DGEMM on Fermi GPU. In: Proceedings of 2011 International Conference for High Performance Computing, Networking, Storage and Analysis, pp. 35–46. Association for Computing Machinery, New York (2011)

    Google Scholar 

  39. Ţăpuş, C., Chung, I., Hollingsworth, J.: Active harmony: towards automated performance tuning. In: Proceedings of the ACM/IEEE Conference on Supercomputing, pp. 1–11. IEEE Computer Society Press, Washington, DC (2002)

    Google Scholar 

  40. Tartara, M., Reghizzi, S.: Continuous learning of compiler heuristics. ACM Trans. Archit. Code Optim. (TACO) 9, 1–25 (2013)

    Article  Google Scholar 

  41. Towns, J., et al.: XSEDE: accelerating scientific discovery. Comput. Sci. Eng. 16, 62–74 (2014)

    Article  Google Scholar 

  42. Vaz, A.I.F., Vicente, L.N.: A particle swarm pattern search method for bound constrained global optimization. J. Glob. Optim. 39, 197–219 (2007). https://doi.org/10.1007/s10898-007-9133-5

    Article  MathSciNet  MATH  Google Scholar 

  43. Vuduc, R., Demmel, J., Yelick, K.: OSKI: a library of automatically tuned sparse matrix kernels. J. Phys: Conf. Ser. 16, 521–530 (2005)

    Google Scholar 

  44. Whaley, R., Petitet, A., Dongarra, J.: Automated empirical optimizations of software and the ATLAS project. Parallel Comput. 27, 3–35 (2001)

    Article  Google Scholar 

  45. Yuki, T., Pouchet, L.N.: PolyBench/C 4.2.1. https://www.cs.colostate.edu/~pouchet/software/polybench/polybench-fortran.html

Download references

Author information

Authors and Affiliations

  1. Department of Chemical Engineering, Carnegie Mellon University, Pittsburgh, PA, 15213, USA

    Benjamin Sauk

  2. H. Milton Stewart School of Industrial and Systems Engineering and School of Chemical and Biomolecular Engineering, Georgia Institute of Technology, Atlanta, GA, 30332, USA

    Nikolaos V. Sahinidis

Authors
  1. Benjamin Sauk
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Nikolaos V. Sahinidis
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Nikolaos V. Sahinidis .

Editor information

Editors and Affiliations

  1. SBA Research, Vienna, Austria

    Dimitris E. Simos

  2. University of Florida, Gainesville, FL, USA

    Panos M. Pardalos

  3. Wilfrid Laurier University, Waterloo, ON, Canada

    Ilias S. Kotsireas

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Sauk, B., Sahinidis, N.V. (2021). HybridTuner: Tuning with Hybrid Derivative-Free Optimization Initialization Strategies. In: Simos, D.E., Pardalos, P.M., Kotsireas, I.S. (eds) Learning and Intelligent Optimization. LION 2021. Lecture Notes in Computer Science(), vol 12931. Springer, Cham. https://doi.org/10.1007/978-3-030-92121-7_29

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-92121-7_29

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-92120-0

  • Online ISBN: 978-3-030-92121-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation