Skip to main content
SpringerLink
Log in

Formal Specifications and Software Testing, a Fruitful Convergence

  • Conference paper
  • First Online:
Formal Methods. FM 2019 International Workshops (FM 2019)

Part of the book series: Lecture Notes in Computer Science ((LNPSE,volume 12233))

Included in the following conference series:

Abstract

This paper gives some account of the evolution of ideas and the main advances in the domain of software testing based on formal specifications and reports some personal anecdotes on my activity in this field. Going back to the seventies, being slightly caricatural, software testing was perceived, on the one hand, by its actors as an empirical activity that had nothing to gain from formal methods, on the other hand, by the advocates of these methods as doomed to disappear based on the belief that in the long run programs will be correct by construction. Currently, these two communities haven’t yet reached a complete consensus. But fortunately there have been some significant moves from both sides and various success stories that allow saying that there is a fruitful convergence toward testing methods based on formal specifications.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    This note can be seen at https://www.cs.utexas.edu/users/EWD/ where it has no date. But one can bracket it between two dated EWDs: EWD 292, 31 Aug 1970, and EWD 306, 16th March 1971 (thanks to Jeremy Gibbons for the hint).

  2. 2.

    During my stay there (1981–1983) I interacted with a group of engineers who developed the software of a telephone switching system. I was impressed by their professionalism. They motivated me to pursue this line of research.

    A (not so funny) anecdote is that some years later, being back as a professor in a university, I visited the same place with a Ph.D. student, searching for challenging case studies. Mood and people had changed and the head of the group explained that their goal was to be first on the market and their development strategy was “quick an dirty”. To that the Ph.D. student replied that “dirty we can do, but quick I am not sure”. The meeting was unproductive...

  3. 3.

    This is similar to the issue of lifting computational types and values to the logical level in Hoare’s logic.

  4. 4.

    IWPTS: 1983–1996; IWTCS: 1997–1999, TESTCOM: 2000–2009, ICTSS: 2010-now.

  5. 5.

    This series of workshops (Formal Approaches to Testing of Software) took place in 2001–2007.

  6. 6.

    A notable omission here is the corpus of research on testing based on FSM (Finite State Machines), which has been considerably influential since the sixties both in hardware and in software testing. For an excellent survey with some historical indications see [28].

References

  1. Barnett, M., Grieskamp, W., Nachmanson, L., Schulte, W., Tillmann, N., Veanes, M.: Towards a tool environment for model-based testing with AsmL. In: Petrenko, A., Ulrich, A. (eds.) FATES 2003. LNCS, vol. 2931, pp. 252–266. Springer, Heidelberg (2004). https://doi.org/10.1007/978-3-540-24617-6_18

    Chapter  Google Scholar 

  2. Bernot, G., Gaudel, M.C., Marre, B.: Software testing based on formal specifications: a theory and a tool. Softw. Eng. J. 6(6), 387–405 (1991)

    Article  Google Scholar 

  3. Bougé, L.: Modeling the notion of program testing; application to test set generation. Theses, Université Pierre et Marie Curie - Paris VI, October 1982. https://tel.archives-ouvertes.fr/tel-00416558

  4. Bougé, L., Choquet, N., Fribourg, L., Gaudel, M.C.: Application of PROLOG to test sets generation from algebraic specifications. In: Ehrig, H., Floyd, C., Nivat, M., Thatcher, J. (eds.) TAPSOFT 1985. LNCS, vol. 186, pp. 261–275. Springer, Heidelberg (1985). https://doi.org/10.1007/3-540-15199-0_17

    Chapter  Google Scholar 

  5. Brinksma, E.: A theory for the derivation of tests. In: Proceedings of 8th International Conference on Protocol Specification, Testing and Verification, pp. 63–74. North-Holland (1988)

    Google Scholar 

  6. Brucker, A.D., Wolff, B.: On theorem prover-based testing. Formal Asp. Comput. 25(5), 683–721 (2013)

    Article  MathSciNet  Google Scholar 

  7. Cavalcanti, A., Gaudel, M.-C.: Testing for refinement in CSP. In: Butler, M., Hinchey, M.G., Larrondo-Petrie, M.M. (eds.) ICFEM 2007. LNCS, vol. 4789, pp. 151–170. Springer, Heidelberg (2007). https://doi.org/10.1007/978-3-540-76650-6_10

    Chapter  Google Scholar 

  8. Cavalcanti, A., Gaudel, M.C.: Testing for refinement in Circus. Acta Inf. 48(2), 97–147 (2011)

    Article  MathSciNet  Google Scholar 

  9. CCITT: Functional specification and description language (SDL), Recommendation Z.100–Z.104 (1984)

    Google Scholar 

  10. Chaudron, M.R.V., Tretmans, J., Wijbrans, K.: Lessons from the application of formal methods to the design of a storm surge barrier control system. In: Wing, J.M., Woodcock, J., Davies, J. (eds.) FM 1999. LNCS, vol. 1709, pp. 1511–1526. Springer, Heidelberg (1999). https://doi.org/10.1007/3-540-48118-4_30

  11. Dauchy, P., Gaudel, M.C., Marre, B.: Using algebraic specifications in software testing: a case study on the software of an automatic subway. J. Syst. Softw. 21(3), 229–244 (1993)

    Article  Google Scholar 

  12. DeMillo, R.A.: Software testing. In: Encyclopedia of Computer Science, pp. 1645–1649. John Wiley and Sons Ltd., GBR (2003)

    Google Scholar 

  13. DeMillo, R.A., Upton, R.J., Perlis, A.J.: Social processes and proofs of theorems and programs. Math. Intell. 3(1), 31–40 (1980). https://doi.org/10.1007/BF03023394

    Article  MathSciNet  MATH  Google Scholar 

  14. Dick, J., Faivre, A.: Automating the generation and sequencing of test cases from model-based specifications. In: Woodcock, J.C.P., Larsen, P.G. (eds.) FME 1993. LNCS, vol. 670, pp. 268–284. Springer, Heidelberg (1993). https://doi.org/10.1007/BFb0024651

    Chapter  Google Scholar 

  15. Dijkstra, E.W.: The humble programmer. Commun. ACM 15(10), 859–866 (1972)

    Article  Google Scholar 

  16. Dubois, C., Giorgetti, A.: Tests and proofs for custom data generators. Formal Aspects Comput. 30(6), 659–684 (2018). https://doi.org/10.1007/s00165-018-0459-1

    Article  MathSciNet  MATH  Google Scholar 

  17. Feliachi, A., Gaudel, M.-C., Wenzel, M., Wolff, B.: The Circus testing theory revisited in Isabelle/HOL. In: Groves, L., Sun, J. (eds.) ICFEM 2013. LNCS, vol. 8144, pp. 131–147. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-41202-8_10

    Chapter  Google Scholar 

  18. Fraser, G., Wotawa, F., Ammann, P.: Issues in using model checkers for test case generation. J. Syst. Softw. 82(9), 1403–1418 (2009)

    Article  Google Scholar 

  19. Gannon, J.D., McMullin, P.R., Hamlet, R.G.: Data-abstraction implementation, specification, and testing. ACM Trans. Program. Lang. Syst. 3(3), 211–223 (1981)

    Article  Google Scholar 

  20. Gaudel, M.-C., Le Gall, P.: Testing data types implementations from algebraic specifications. In: Hierons, R.M., Bowen, J.P., Harman, M. (eds.) Formal Methods and Testing. LNCS, vol. 4949, pp. 209–239. Springer, Heidelberg (2008). https://doi.org/10.1007/978-3-540-78917-8_7

    Chapter  Google Scholar 

  21. Gaudel, M.C., James, P.R.: Testing algebraic data types and processes: a unifying theory. Formal Asp. Comput. 10(5–6), 436–451 (1998)

    Article  Google Scholar 

  22. Hierons, R.M., et al.: Using formal specifications to support testing. ACM Comput. Surv. 41(2), 9:1–9:76 (2009)

    Google Scholar 

  23. Hoare, C.A.R.: How did software get so reliable without proof? In: Gaudel, M.-C., Woodcock, J. (eds.) FME 1996. LNCS, vol. 1051, pp. 1–17. Springer, Heidelberg (1996). https://doi.org/10.1007/3-540-60973-3_77

    Chapter  Google Scholar 

  24. Houssais, B.: Verification of an Algol 68 implementation. ACM SIGPLAN Not. 12(6), 117–128 (1977)

    Article  Google Scholar 

  25. ISO: Conformance testing methodology and framework. International Standard IS-9646 (1991)

    Google Scholar 

  26. Jalote, P.: Specification and testing of abstract data types. In: IEEE International Computer Software and Applications Conference COMSAC, pp. 508–511 (1983)

    Google Scholar 

  27. Jard, C., Jéron, T.: TGV: theory, principles and algorithms. Int. J. Softw. Tools Technol. Transf. 7(4), 297–315 (2005)

    Article  Google Scholar 

  28. Lee, D., Yannakakis, M.: Principles and methods of testing finite state machines-a survey. Proc. IEEE 84(8), 1090–1123 (1996)

    Article  Google Scholar 

  29. Legeard, B., Peureux, F.: Generation of functional test sequences from B formal specifications-presentation and industrial case study. In: 16th IEEE International Conference on Automated Software Engineering (ASE 2001), Coronado Island, San Diego, CA, USA, 26–29 November 2001, pp. 377–381. IEEE Computer Society (2001)

    Google Scholar 

  30. Marre, B., Blanc, B.: Test selection strategies for Lustre descriptions in GATeL. Electr. Notes Theor. Comput. Sci. 111, 93–111 (2005)

    Article  Google Scholar 

  31. Petiot, G., Kosmatov, N., Botella, B., Giorgetti, A., Julliand, J.: How testing helps to diagnose proof failures. Formal Aspects Comput. 30(6), 629–657 (2018). https://doi.org/10.1007/s00165-018-0456-4

    Article  MathSciNet  MATH  Google Scholar 

  32. Pitt, D.H., Freestone, D.: The derivation of conformance tests from LOTOS specifications. IEEE Trans. Software Eng. 16(12), 1337–1343 (1990)

    Article  Google Scholar 

  33. Sarikaya, B., von Bochmann, G.: Some experience with test sequence generation for protocols. In: Protocol Specification, Testing and Verification, Proceedings of the IFIP WG6.1 Second International Workshop on Protocol Specification, Testing and Verification, Idyllwild, CA, USA, 17–20 May 1982, pp. 555–567. North-Holland (1982)

    Google Scholar 

  34. Satpathy, M., Butler, M., Leuschel, M., Ramesh, S.: Automatic testing from formal specifications. In: Gurevich, Y., Meyer, B. (eds.) TAP 2007. LNCS, vol. 4454, pp. 95–113. Springer, Heidelberg (2007). https://doi.org/10.1007/978-3-540-73770-4_6

    Chapter  Google Scholar 

  35. Stocks, P., Carrington, D.A.: Test templates: a specification-based testing framework. In: Proceedings of the 15th International Conference on Software Engineering, Baltimore, Maryland, USA, 17–21 May 1993, pp. 405–414. IEEE Computer Society/ACM Press (1993)

    Google Scholar 

  36. Tretmans, J.: Test generation with inputs, outputs and repetitive quiescence. Softw. Concepts Tools 17(3), 103–120 (1996)

    MATH  Google Scholar 

  37. Tretmans, J., van de Laar, P.: Model-based testing with TorXakis. In: Proceedings of 30th CECIIS, the Central European Conference on Information and Intelligent Systems, Varaždin, Croatia, 2–4 October 2019, pp. 247–258 (1987)

    Google Scholar 

  38. Ural, H.: A test derivation method for protocol conformance testing. In: Protocol Specification, Testing and Verification VII, Proceedings of the IFIP WG6.1 Seventh International Conference on Protocol Specification, Testing and Verification, Zurich, Switzerland, 5–8 May 1987, pp. 347–358. North-Holland (1987)

    Google Scholar 

  39. Veanes, M., Campbell, C., Grieskamp, W., Schulte, W., Tillmann, N., Nachmanson, L.: Model-based testing of object-oriented reactive systems with Spec Explorer. In: Hierons, R.M., Bowen, J.P., Harman, M. (eds.) Formal Methods and Testing. LNCS, vol. 4949, pp. 39–76. Springer, Heidelberg (2008). https://doi.org/10.1007/978-3-540-78917-8_2

    Chapter  MATH  Google Scholar 

Download references

Acknowledgment

I am grateful to Burkhart Wolff and the members of the LRI Test Club who gave me the idea to talk and write on this topic.

Author information

Authors and Affiliations

  1. LRI, Université Paris-Sud, Orsay, France

    Marie-Claude Gaudel

Authors
  1. Marie-Claude Gaudel
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Marie-Claude Gaudel .

Editor information

Editors and Affiliations

  1. McMaster University, Hamilton, ON, Canada

    Emil Sekerinski

  2. University of Porto, Porto, Portugal

    Nelma Moreira

  3. University of Minho, Braga, Portugal

    José N. Oliveira

  4. Argo Ai, Munich, Germany

    Daniel Ratiu

  5. University of Pisa, Pisa, Italy

    Riccardo Guidotti

  6. University of Liverpool, Liverpool, UK

    Marie Farrell

  7. University of Liverpool, Liverpool, UK

    Matt Luckcuck

  8. University of Exeter, Exeter, UK

    Diego Marmsoler

  9. University of Minho, Braga, Portugal

    José Campos

  10. University of Newcastle, Newcastle upon Tyne, UK

    Troy Astarte

  11. Claude Bernard University, Lyon, France

    Laure Gonnord

  12. Nazarbayev University, Nur-Sultan, Kazakhstan

    Antonio Cerone

  13. University of Surrey, Guildford, UK

    Luis Couto

  14. University of Surrey, Guildford, UK

    Brijesh Dongol

  15. University of Giessen, Giessen, Germany

    Martin Kutrib

  16. University of Lisbon, Lisbon, Portugal

    Pedro Monteiro

  17. Airbus Operations S.A.S., Toulouse, France

    David Delmas

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Gaudel, MC. (2020). Formal Specifications and Software Testing, a Fruitful Convergence. In: Sekerinski, E., et al. Formal Methods. FM 2019 International Workshops. FM 2019. Lecture Notes in Computer Science(), vol 12233. Springer, Cham. https://doi.org/10.1007/978-3-030-54997-8_5

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-54997-8_5

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-54996-1

  • Online ISBN: 978-3-030-54997-8

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation