Skip to main content
SpringerLink
Log in

Risk-based testing via active continuous quality control

  • RBT
  • Published:
International Journal on Software Tools for Technology Transfer Aims and scope Submit manuscript

Abstract

In this paper, we show how our approach to active continuous quality control (ACQC), which employs learning technology to automatically maintain test models along the whole life cycle, can be extended to include risk analysts for supporting risk-based testing. Key to this enhancement is the tailoring of ACQC’s characteristic automata learning-based model extraction to prioritize critical aspects. Technically, risk analysts are provided with an abstract modeling level tailored to design test components (learning symbols) that encompass data-flow constraints reflecting a given risk profile. The resulting alphabet models are already sufficient to steer the ACQC process in a fashion that increases the risk coverage, while it at the same time radically reduces the testing effort. We illustrate our approach by means of case studies with Springer’s Online Conference Service (OCS) which show the impact of the risk prioritization on the performance: risk-based regression testing tailored for system migration and for pure functional evolution.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19

Similar content being viewed by others

Notes

  1. This case study of the router with its over 20.000 states was a first indication of the scalability of learning-based validation.

  2. Java Platform, Enterprise Edition. For more information please see http://www.oracle.com/technetwork/java/javaee/.

  3. The Java Platform, Enterprise Edition 6 was released in December 2009.

  4. In the active automata learning context, the term system under learning (SUL) is used.

  5. The executions on the SUT are denoted by test cases, the corresponding models are denoted by test case models, and the abstract input word is denoted by membership query.

  6. LearnLib is available at http://www.learnlib.de.

  7. From now on we do not consider ‘maximal reuse’ as we realized it on the (learn-) algorithm level by reusing the distinguishing suffixes. This way we avoid equivalence queries as we switch to a new application version of the system under learning. Fortunately, this feature is implemented once and may be used for every incremental learn setup.

  8. CDI is defined in JSR 299, for a short introduction please see http://www.oracle.com/technetwork/articles/javaee/javaee6overview-141808.html#webbeans.

  9. For the first input symbol the ‘reset’ activity is selected as predecessor since the ‘reset’ is already connected to the start activity (cf. Fig. 6).

  10. http://tapestry.apache.org.

  11. Before the bidding and assignment phase started only after the upload phase had been ended.

  12. A corresponding check had been unnecessary, because previously the upload phase had to be terminated before the assignment phase could be started.

References

  1. Aarts, F., Jonsson, F., Uijen, J.: Generating models of infinite-state communication protocols using regular inference with abstraction. In: ICTSS ’10, Volume 6435 of LNCS, pp. 188–204. Springer, Berlin (2010)

  2. Aarts, F., Schmaltz, J., Vaandrager, F.: Inference and abstraction of the biometric passport. In: ISoLA 2010, Volume 6415 of LNCS, pp. 673–686. Springer, Berlin (2010)

  3. Angluin, D.: Learning regular sets from queries and counterexamples. Inf. Comput. 75(2), 87–106 (1987)

    Article  MATH  MathSciNet  Google Scholar 

  4. Bach, J.: Heuristic risk-based testing. Softw. Test. Quality Eng. Mag. 11, 99 (1999)

    Google Scholar 

  5. Bauer, O., Neubauer, J., Steffen, B., Howar, F.: Reusing system states by active learning algorithms. In: Moschitti, A., Scandariato, R. (eds.) Eternal Systems, Volume 255 of CCSE. Springer, Berlin (2012)

    Google Scholar 

  6. Berg, T., Grinchtein, O., Jonsson, B., Leucker, M., Raffelt, H., Steffen, B.: On the correspondence between conformance testing and regular inference. In: FASE ’05, Volume 3442 of LNCS, pp. 175–189. Springer, Berlin (2005)

  7. Beydeda, S., Gruhn, V.: Integrating white- and black-box techniques for class-level regression testing. In: COMPSAC ’01, Washington, DC, USA, pp. 357–362. IEEE Computer Society (2001)

  8. Bossert, V., Hiet, G., Henin, G.: Modelling to simulate botnet command and control protocols for the evaluation of network intrusion detection systems. In: SAR-SSI ’11, pp. 1–8. IEEE Computer Society (2011)

  9. Chen, Y., Probert, R.L., Ural, H.: Model-based regression test suite generation using dependence analysis. In: A-MOST ’07, New York, NY, USA, pp. 54–62. ACM (2007)

  10. Chow, T.S.: Testing software design modeled by finite-state machines. IEEE Trans. Softw. Eng. 4(3), 178–187 (1978)

    Article  MATH  Google Scholar 

  11. Felderer, M., Haisjackl, M., Breu, R., Motz, J.: Integrating manual and automatic risk assessment for risk-based testing. In: Biffl, S., Winkler, D., Bergsmann, J. (eds.) Software Quality, Process Automation in Software Development, Volume 94 of Lecture Notes in Business Information Processing, pp. 159–180. Springer, Berlin (2012)

    Chapter  Google Scholar 

  12. Felderer, M., Ramler, R.: A multiple case study on risk-based testing in industry. sttt-RBT, Under, Review (2014)

  13. Fowler, M.: Inversion of control containers and the dependency injection pattern (2004) . http://www.martinfowler.com/articles/injection.html

  14. Gerrard, P., Thompson, N.: Risk based e-business resting. Artech House (2002)

  15. Gnesi, S., Margaria, T.: Formal Methods for Industrial Critical Systems: A Survey of Applications. Wiley, :London (2012)

    Book  Google Scholar 

  16. Grabowski, J., Hogrefe, D., Réthy, G., Schieferdecker, I., Wiles, A., Willcock, C.: An introduction to the testing and test control notation (ttcn-3). Comput. Netw. 42(3), 375–403 (2003)

    Article  MATH  Google Scholar 

  17. Grabowski, J., Wiles, A., Willcock, C., Hogrefe, D.: On the design of the new testing language ttcn-3. In: Ural, H., Probert, R., Bochmann, G. (eds.) Testing of Communicating Systems, Volume 48 of IFIP Advances in Information and Communication Technology, pp. 161–176. Springer, New York (2000)

    Google Scholar 

  18. Grossmann, J.: Testing hybrid systems with ttcn-3 embedded. sttt-RBT. Under Review (2014)

  19. Hagerer, A. Hungar, H., Niese, O., Steffen, B.: Model generation by moderated regular extrapolation. In FASE ’02, pp. 80–95, London, UK. Springer, Berlin (2002)

  20. Howar, F., Steffen, B., Jonsson, B., Cassel, S.: Inferring canonical register automata. In: Kuncak, V., Rybalchenko, A. (eds.) Verification, Model Checking, and Abstract Interpretation, Volume 7148 of Lecture Notes in Computer Science, pp. 251–266. Springer, Berlin (2012)

    Google Scholar 

  21. Howar, F., Steffen, B., Merten, M.: Automata learning with automated alphabet abstraction refinement. In: Verification, Model Checking, and Abstract Interpretation, Volume 6538 of LNCS, pp. 263–277. Springer, Berlin (2011)

  22. Hungar, H., Niese, O., Steffen, B.: Domain-specific optimization in automata learning. In: Computer Aided Verification, Volume 2725 of LNCS, pp. 315–327. Springer, Berlin (2003)

  23. Jörges, S., Margaria, T., Steffen, B.: Genesys: service-oriented construction of property conform code generators. ISSE 4(4), 361–384 (2008)

    Google Scholar 

  24. Jéron, T.: Symbolic model-based test selection. Electron. Notes Theor. Comput. Sci. 240(0):167–184 (2009). Proceedings of the Eleventh Brazilian Symposium on Formal Methods (SBMF 2008)

  25. Jörges, S.: Construction and Evolution of Code Generators: A Model-Driven and Service-Oriented Approach. Springer, Berlin (2013)

    Book  Google Scholar 

  26. Karusseit, M., Margaria, T.: Feature-based modelling of a complex, online-reconfigurable decision support service. In WWV ’05, March 2005. ENTCS 1132

  27. Korel, B., Al-Yami, A.M.: Automated regression test generation. SIGSOFT Softw. Eng. Notes 23(2), 143–152 (1998)

    Article  Google Scholar 

  28. Lund, M.S., Solhaug, B.: Model-Driven Risk Analysis. Springer, Berlin (2011)

    Book  Google Scholar 

  29. Marchand, H., Dubreil, J., Jéron, T.: Automatic testing of access control for security properties. In: Núñez, M., Baker, P., Merayo, M. (eds.) Testing of Software and Communication Systems, Volume 5826 of Lecture Notes in Computer Science, pp. 113–128. Springer, Berlin (2009)

    Google Scholar 

  30. Margaria, T., Niese, O., Raffelt, H., Steffen. B.: Efficient test-based model generation for legacy reactive systems. In: HLDVT ’04, Washington, DC, USA, pp. 95–100. IEEE Computer Society (2004)

  31. Margaria, T., Steffen, B.: Lightweight coarse-grained coordination: a scalable system-level approach. STTT 5(2–3), 107–123 (2004)

  32. Margaria, T., Steffen, B.: Agile IT: thinking in user-centric models. In: ISoLA 2009, Volume 17 of Communications in Computer and Information Science, pp. 490–502. Springer, Berlin (2009)

  33. Margaria, T., Steffen, B.: Service-orientation: conquering complexity with XMDD. In: Hinchey, M., Coyle, L. (eds.) Conquering Complexity, pp. 217–236. Springer, London (2012)

    Chapter  Google Scholar 

  34. Merten, M., Steffen, B., Howar, F., Margaria T.: Next generation learnlib. In: TACAS 2011, Volume 6605 of LNCS, pp. 220–223. Springer, Berlin (2011)

  35. Müller-Olm, M., Schmidt, D., Steffen, B.: Model-checking: a tutorial introduction. In: Cortesi, A., Filé, G. (eds.) Static Analysis, Volume 1694 of LNCS, pp. 330–354. Springer, Berlin (1999)

  36. Neubauer, J., Steffen, B.: Plug-and-play higher-order process integration. Computer 46(11), 56–62 (2013)

    Article  Google Scholar 

  37. Neubauer, J., Steffen, B.: Learning-based cross-platform conformance testing . STVR. Wiley (2014)

  38. Neubauer, J. Steffen, B., Margaria, T.: Higher-order process modeling: product-lining, variability modeling and beyond. arXiv, preprint arXiv:1309.5143 (2013)

  39. Pickin, S., Jard, C., Jéron, T., Jézéquel, J.-M., Traon, Y.L.: Test synthesis from uml models of distributed software. IEEE Trans. Softw. Eng. 33(4), 252–269 (2007)

    Article  Google Scholar 

  40. Raffelt, H., Merten, M., Steffen, B., Margaria, T.: Dynamic testing via automata learning. STTT 11(4), 307–324 (2009)

    Article  Google Scholar 

  41. Raffelt, H., Steffen, B., Berg, T., Margaria, T.: LearnLib: a framework for extrapolating behavioral models. Int. J. Softw. Tools Technol. Transf. 11(5), 393–407 (2009)

    Article  Google Scholar 

  42. Rivest, R.L., Schapire, R.E.: Inference of finite automata using homing sequences. Inf. Comput. 103(2), 299–347 (1993)

    Article  MATH  MathSciNet  Google Scholar 

  43. Rosenberg, L., Stapko, R., Gallo. A.: Risk-based object oriented testing. In: Proceedings of 13th International Software/Internet Quality Week-QW, San Francisco, California, USA, Vol. 2 (2000)

  44. Rothermel, G., Harrold, M.J., Dedhia, J.: Regression test selection for C++ software. Softw. Test. Verif. Reliab. 10, 2000 (1999)

    Google Scholar 

  45. Shahbaz, M., Shashidhar, K.C., Eschbach. R.: Iterative refinement of specification for component based embedded systems. In: ISSTA ’11, pp. 276–286. ACM (2011)

  46. Smolka, S., Steffen, B.: Priority as extremal probability. Formal Asp. Comput. 8(5), 585–606 (1996)

  47. Steffen, B.: Characteristic formulae. In: ICALP, Volume 372 of LNCS, pp. 723–732 (1989)

  48. Steffen, B., Howar, F., Merten. M.: Introduction to active automata learning from a practical perspective. In: Bernardo, M., Issarny, V. (eds.) Formal Methods for Eternal Networked Software Systems, Volume 6659 of LNCS, pp. 256–296. Springer, Berlin (2011)

  49. Steffen, B., Margaria, T., Nagel, R., Jörges, S., Kubczak, C.: Model-driven development with the jABC. In: Bin, E., Ziv, A., Ur, S. (eds.) Hardware and Software. Verification and Testing, Volume 4383 of Lecture Notes in Computer Science, pp. 92–108. Springer, Berlin (2007)

    Google Scholar 

  50. Stepien, B., Peyton, L.: Innovation and evolution in integrated web application testing with ttcn-3. sttt-RBT. Under Review (2014)

  51. Tretmans, J.: Model based testing with labelled transition systems. In: Hierons, R.M., Bowen, J.P., Harman, M. (eds.) Formal Methods and Testing, Volume 4949 of LNCS, pp. 1–38. Springer, Berlin (2008)

  52. Tretmans, J.: Model-based testing and some steps towards test-based modelling. In: Bernardo, M., Issarny, V. (eds.) Formal Methods for Eternal Networked Software Systems, Volume 6659 of LNCS, pp. 297–326. Springer, Berlin (2011)

    Chapter  Google Scholar 

  53. Utting, M., Pretschner, A., Legeard, B.: A taxonomy of model-based testing approaches. Softw. Test. Verif. Reliab. 22(5), 297–312 (2012)

    Article  Google Scholar 

  54. Windmüller, S., Neubauer, J., Steffen, B., Howar, F., Bauer, O.: Active continuous quality control. In: Proceedings of the 16th International ACM Sigsoft symposium on Component-Based Software Engineering, CBSE ’13, New York, NY, USA, pp. 111–120. ACM (2013)

  55. Xu, L., Dias, M., Richardson, D.: Generating regression tests via model checking. In: COMPSAC ’04, Washington, DC, USA, pp. 336–341. IEEE Computer Society (2004)

Download references

Author information

Authors and Affiliations

  1. TU Dortmund University, Dortmund, Germany

    Johannes Neubauer, Stephan Windmüller & Bernhard Steffen

Authors
  1. Johannes Neubauer
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Stephan Windmüller
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Bernhard Steffen
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Johannes Neubauer.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Neubauer, J., Windmüller, S. & Steffen, B. Risk-based testing via active continuous quality control. Int J Softw Tools Technol Transfer 16, 569–591 (2014). https://doi.org/10.1007/s10009-014-0321-6

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10009-014-0321-6

Keywords

Search

Navigation