Abstract
Software verification tools automatically prove the correctness of programs with respect to user supplied specifications. Today, such tools implement a range of different types of analyses. As different analyses are good at different sorts of verification tasks, state-of-the-art tools often employ sequential compositions of analyses in which every analysis gets a fixed time slot assigned for verification. As a consequence, however, one analysis might consume parts of the overall available time although it does not finish within its time slot.
In this paper, we propose timeout prediction as a way to determine when an analysis should get its full time slot and when to prematurely stop it. Our technique for timeout prediction employs machine learning to predict whether a given analysis will terminate on a given verification task (within a time limit) or will time out. To this end, we develop static as well as dynamic features of verification tasks and analyses. Values of static features can be statically determined for tasks; dynamic features are determined while an analysis is already running. Our experimental evaluation shows that we can predict timeouts with a high accuracy.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Beyer, D., Dangl, M.: Strategy selection for software verification based on boolean features. In: Margaria, T., Steffen, B. (eds.) ISoLA 2018. LNCS, vol. 11245, pp. 144–159. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-03421-4_11
Beyer, D., Henzinger, T.A., Théoduloz, G.: Configurable software verification: concretizing the convergence of model checking and program analysis. In: Damm, W., Hermanns, H. (eds.) CAV 2007. LNCS, vol. 4590, pp. 504–518. Springer, Heidelberg (2007). https://doi.org/10.1007/978-3-540-73368-3_51
Beyer, D., Kanav, S., Richter, C.: Construction of verifier combinations based on off-the-shelf verifiers. In: FASE 2022. LNCS, vol. 13241, pp. 49–70. Springer, Cham (2022). https://doi.org/10.1007/978-3-030-99429-7_3
Beyer, D., Keremoglu, M.E.: CPAchecker: a tool for configurable software verification. In: Gopalakrishnan, G., Qadeer, S. (eds.) CAV 2011. LNCS, vol. 6806, pp. 184–190. Springer, Heidelberg (2011). https://doi.org/10.1007/978-3-642-22110-1_16
Beyer, D., Keremoglu, M.E., Wendler, P.: Predicate abstraction with adjustable-block encoding. In: FMCAD 2010, pp. 189–197. IEEE (2010)
Beyer, D., Löwe, S., Wendler, P.: Reliable benchmarking: requirements and solutions. STTT 21(1), 1–29 (2019)
Chalupa, M., Henzinger, T.A.: BUBAAK: runtime monitoring of program verifiers. In: Sankaranarayanan, S., Sharygina, N. (eds.) Tools and Algorithms for the Construction and Analysis of Systems. TACAS 2023. LNCS, vol. 13994, pp. 535–540. Springer, Cham (2023). https://doi.org/10.1007/978-3-031-30820-8_32
Chalupa, M., Mihalkovič, V., Řechtáčková, A., Zaoral, L., Strejček, J.: Symbiotic 9: string analysis and backward symbolic execution with loop folding. In: TACAS 2022. LNCS, vol. 13244, pp. 462–467. Springer, Cham (2022). https://doi.org/10.1007/978-3-030-99527-0_32
Chalupa, M., Strejcek, J., Vitovská, M.: Joint forces for memory safety checking revisited. Int. J. Softw. Tools Technol. Transf. 22(2), 115–133 (2020). https://doi.org/10.1007/s10009-019-00526-2
Clarke, E., Grumberg, O., Jha, S., Lu, Y., Veith, H.: Counterexample-guided abstraction refinement. In: Emerson, E.A., Sistla, A.P. (eds.) CAV 2000. LNCS, vol. 1855, pp. 154–169. Springer, Heidelberg (2000). https://doi.org/10.1007/10722167_15
Cousot, P., Cousot, R.: Abstract interpretation: a unified lattice model for static analysis of programs by construction or approximation of fixpoints. In: ACM 1977, pp. 238–252. ACM (1977). https://doi.org/10.1145/512950.512973
Czech, M., Hüllermeier, E., Jakobs, M., Wehrheim, H.: Predicting rankings of software verification tools. In: SWAN@ESEC/SIGSOFT FSE 2017, pp. 23–26. ACM (2017). https://doi.org/10.1145/3121257.3121262
Dangl, M., Löwe, S., Wendler, P.: CPAchecker with support for recursive programs and floating-point arithmetic. In: Baier, C., Tinelli, C. (eds.) TACAS 2015. LNCS, vol. 9035, pp. 423–425. Springer, Heidelberg (2015). https://doi.org/10.1007/978-3-662-46681-0_34
Demyanova, Y., Pani, T., Veith, H., Zuleger, F.: Empirical software metrics for benchmarking of verification tools. Formal Methods Syst. Des. 50(2), 289–316 (2017). https://doi.org/10.1007/s10703-016-0264-5
Ernst, G.: KORN–software verification with horn clauses (Competition Contribution). In: Sankaranarayanan, S., Sharygina, N. (eds) Tools and Algorithms for the Construction and Analysis of Systems. TACAS 2023. LNCS, vol. 13994, pp 559–564. Springer, Cham (2023). https://doi.org/10.1007/978-3-031-30820-8_36
Ernst, G., Huisman, M., Mostowski, W., Ulbrich, M.: VerifyThis – verification competition with a human factor. In: Beyer, D., Huisman, M., Kordon, F., Steffen, B. (eds.) TACAS 2019. LNCS, vol. 11429, pp. 176–195. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-17502-3_12
Graf, S., Saidi, H.: Construction of abstract state graphs with PVS. In: Grumberg, O. (ed.) CAV 1997. LNCS, vol. 1254, pp. 72–83. Springer, Heidelberg (1997). https://doi.org/10.1007/3-540-63166-6_10
Hajdu, Á., Micskei, Z.: Efficient strategies for CEGAR-based model checking. J. Autom. Reason. 64(6), 1051–1091 (2020). https://doi.org/10.1007/s10817-019-09535-x
Healy, A., Monahan, R., Power, J.F.: Predicting SMT solver performance for software verification. In: F-IDE@FM 2016, EPTCS, vol. 240, pp. 20–37 (2016). https://doi.org/10.4204/EPTCS.240.2
Heizmann, M., Hoenicke, J., Podelski, A.: Software model checking for people who love automata. In: Sharygina, N., Veith, H. (eds.) CAV 2013. LNCS, vol. 8044, pp. 36–52. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-39799-8_2
Leeson, W., Dwyer, M.B.: Graves-CPA: a graph-attention verifier selector (Competition Contribution). In: TACAS 2022. LNCS, vol. 13244, pp. 440–445. Springer, Cham (2022). https://doi.org/10.1007/978-3-030-99527-0_28
Leeson, W., Dwyer, M.B., Filieri, A.: Sibyl: improving software engineering tools with SMT selection. In: Proceedings of ICSE (2023)
Luckow, K., et al.: JDart: a dynamic symbolic analysis framework. In: Chechik, M., Raskin, J.-F. (eds.) TACAS 2016. LNCS, vol. 9636, pp. 442–459. Springer, Heidelberg (2016). https://doi.org/10.1007/978-3-662-49674-9_26
Luo, S., Xu, H., Bi, Y., Wang, X., Zhou, Y.: Boosting symbolic execution via constraint solving time prediction (experience paper). In: ISSTA 2021, pp. 336–347. ACM (2021). https://doi.org/10.1145/3460319.3464813
Richter, C., Hüllermeier, E., Jakobs, M., Wehrheim, H.: Algorithm selection for software validation based on graph kernels. Autom. Softw. Eng. 27(1), 153–186 (2020). https://doi.org/10.1007/s10515-020-00270-x
Richter, C., Wehrheim, H.: PeSCo: predicting sequential combinations of verifiers. In: Beyer, D., Huisman, M., Kordon, F., Steffen, B. (eds.) TACAS 2019. LNCS, vol. 11429, pp. 229–233. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-17502-3_19
Richter, C., Wehrheim, H.: Attend and represent: a novel view on algorithm selection for software verification. In: ASE 2020, pp. 1016–1028. IEEE (2020). https://doi.org/10.1145/3324884.3416633
Saan, S. et al.: GOBLINT: autotuning thread-modular abstract interpretation. In: Sankaranarayanan, S., Sharygina, N. (eds.) Tools and Algorithms for the Construction and Analysis of Systems. TACAS 2023. LNCS, vol. 13994, pp. 547–552. Springer, Cham (2023). https://doi.org/10.1007/978-3-031-30820-8_34
Scott, J., Niemetz, A., Preiner, M., Nejati, S., Ganesh, V.: MachSMT: a machine learning-based algorithm selector for SMT solvers. In: TACAS 2021. LNCS, vol. 12652, pp. 303–325. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-72013-1_16
Thoben, N.: Online Performance Prediction of Software Verification using Machine Learning. Master’s thesis, University of Oldenburg, Department of Computer Science (2023)
Tulsian, V., Kanade, A., Kumar, R., Lal, A., Nori, A.V.: MUX: algorithm selection for software model checkers. In: MSR 2014, pp. 132–141. ACM (2014). https://doi.org/10.1145/2597073.2597080
Wahl, T.: The k-induction principle (2013). http://www.ccs.neu.edu/home/wahl/Publications/k-induction.pdf
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Thoben, N., Haltermann, J., Wehrheim, H. (2023). Timeout Prediction for Software Analyses. In: Ferreira, C., Willemse, T.A.C. (eds) Software Engineering and Formal Methods. SEFM 2023. Lecture Notes in Computer Science, vol 14323. Springer, Cham. https://doi.org/10.1007/978-3-031-47115-5_19
Download citation
DOI: https://doi.org/10.1007/978-3-031-47115-5_19
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-47114-8
Online ISBN: 978-3-031-47115-5
eBook Packages: Computer ScienceComputer Science (R0)