Skip to main content
SpringerLink
Log in

Mutation Testing for RoboChart

  • Chapter
  • First Online:
Software Engineering for Robotics

  • 2100 Accesses

Abstract

This chapter describes a test-generation approach that takes as input a model S of the expected behavior of a robotic system and seeds faults into S, leading to a set of mutants of S. Given a mutant M of S, we check whether M is a valid implementation of S, and, if it is not, we find a test case that demonstrates this: a test case that reveals the seeded fault. In order to automate this approach, we used the Wodel tool to seed faults and a combination of two tools, RoboTool and FDR, to generate tests that detect the seeded faults. The result is an overall test-generation technique that can be automated and that derives test cases that are guaranteed to find certain faults.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 109.00
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 139.00
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 129.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. H. Agrawal, R.A. DeMillo, B. Hathaway, W. Hsu, W. Hsu, E.W. Krauser, R.J. Martin, A.P. Mathur, E. Spafford, Design of mutant operators for the C programming language. Technical report, Purdue University, 1989

    Google Scholar 

  2. P. Ammann, J. Offutt, Introduction to Software Testing (Cambridge University Press, Cambridge, 2008)

    Book  Google Scholar 

  3. P. Ammann, M.E. Delamaro, J. Offutt, Establishing theoretical minimal sets of mutants, in Verification and Validation 2014 IEEE Seventh International Conference on Software Testing (2014), pp. 21–30 ISSN: 2159–4848

    Google Scholar 

  4. J.H. Andrews, L.C. Briand, Y. Labiche, Is mutation an appropriate tool for testing experiments? in 27th International Conference on Software Engineering (ICSE 2005), 15–21 May 2005, St. Louis, Missouri, ed. by G.-C. Roman, W.G. Griswold, B. Nuseibeh (ACM, New York, 2005), pp. 402–411

    Google Scholar 

  5. J.S. Bradbury, J.R. Cordy, J. Dingel, Mutation operators for concurrent Java (J2SE 5.0), in Second Workshop on Mutation Analysis (Mutation 2006 - ISSRE Workshops 2006) (2006), pp. 83–92

    Google Scholar 

  6. T.A. Budd, D. Angluin, Two notions of correctness and their relation to testing. Acta Inform. 18(1), 31–45 (1982)

    Article  MathSciNet  Google Scholar 

  7. P.C. Cañizares, A. Núñez, M.G. Merayo, Mutomvo: mutation testing framework for simulated cloud and HPC environments. J. Syst. Softw. 143, 187–207 (2018)

    Article  Google Scholar 

  8. A. Cavalcanti, J. Baxter, R.M. Hierons, R. Lefticaru, Testing robots using CSP, in Proceedings of Tests and Proofs - 13th International Conference, TAP 2019, Held as Part of the Third World Congress on Formal Methods 2019, Porto, 9–11 Oct 2019, ed. by D. Beyer, C. Keller. Lecture Notes in Computer Science, vol. 11823 (Springer, Berlin, 2019), pp. 21–38

    Google Scholar 

  9. T.T. Chekam, M. Papadakis, Y.L. Traon, M. Harman, An empirical study on mutation, statement and branch coverage fault revelation that avoids the unreliable clean program assumption, in Proceedings of the 39th International Conference on Software Engineering, ICSE 2017, Buenos Aires, 20–28 May 2017, ed. by S. Uchitel, A. Orso, M.P. Robillard (IEEE/ACM, New York, 2017), pp. 597–608

    Google Scholar 

  10. J.A. Clark, H. Dan, R.M. Hierons, Semantic mutation testing. Sci. Comput. Program. 78(4), 345–363 (2013)

    Article  Google Scholar 

  11. B. Darolti, Software engineering for robotics: an autonomous robotic vacuum cleaner for solar panels. Master’s thesis, University of York, 2019

    Google Scholar 

  12. M.E. Delamaro, J.C. Maldonado, Proteum-a tool for the assessment of test adequacy for C programs, in Proceedings of the Conference on Performability in Computing Systems (PCS’96), New Brunswick, NJ, July 1996, pp. 79–95

    Google Scholar 

  13. M.E. Delamaro, J. Offutt, P. Ammann, Designing deletion mutation operators, in Verification and Validation 2014 IEEE Seventh International Conference on Software Testing (2014), pp. 11–20

    Google Scholar 

  14. P. Delgado-Pérez, I. Medina-Bulo, J.J. Domínguez-Jiménez, A García-Domínguez, F. Palomo-Lozano, Class mutation operators for C++ object-oriented systems. Ann. Télécomm. 70(3–4), 137–148 (2015)

    Google Scholar 

  15. P. Delgado-Pérez, S. Segura, I. Medina-Bulo, Assessment of C++ object-oriented mutation operators: a selective mutation approach. Softw. Test. Verif. Reliab. 27(4–5), e1630 (2017)

    Google Scholar 

  16. R.A. DeMillo, A.J. Offutt, Constraint-based automatic test data generation. IEEE Trans. Softw. Eng. 17(9), 900–910 (1991)

    Article  Google Scholar 

  17. R.A. DeMillo, R.J. Lipton, F.G. Sayward, Hints on test data selection: help for the practicing programmer. Computer 11(4), 34–41 (1978)

    Article  Google Scholar 

  18. R.A. DeMillo, D.S. Guindi, W.M. McCracken, A.J. Offutt, K.N. King, An extended overview of the Mothra software testing environment, in [1988] Proceedings. Second Workshop on Software Testing, Verification, and Analysis, July 1988, pp. 142–151

    Google Scholar 

  19. L. Deng, J. Offutt, N. Li, Empirical evaluation of the statement deletion mutation operator, in Sixth IEEE International Conference on Software Testing, Verification and Validation, ICST 2013, Luxembourg, 18–22 March 2013 (IEEE Computer Society, Washington, 2013), pp. 84–93

    Google Scholar 

  20. A. Estero-Botaro, F. Palomo-Lozano, I. Medina-Bulo, Quantitative evaluation of mutation operators for WS-BPEL compositions, in Proceedings of the 3rd International Conference on Software Testing, Verification, and Validation Workshops (ICSTW’10) (2010), pp. 142–150

    Google Scholar 

  21. S.C.P.F. Fabbri, M.E. Delamaro, J.C. Maldonado, P.C. Masiero, Mutation analysis testing for finite state machines, in Proceedings of the 5th International Symposium on Software Reliability Engineering, Monterey, CA, 6–9 Nov 1994, pp. 220–229

    Google Scholar 

  22. S.C.P.F. Fabbri, J.C. Maldonado, P.C. Masiero, M.E. Delamaro, W.E. Wong, Mutation testing applied to validate specifications based on Petri Nets, in Proceedings of the IFIP TC6 8th International Conference on Formal Description Techniques VIII (1995), pp. 329–337

    Google Scholar 

  23. S.C.P.F. Fabbri, J.C. Maldonado, T. Sugeta, P.C. Masiero, Mutation testing applied to validate specifications based on Statecharts, in Proceedings of the 10th International Symposium on Software Reliability Engineering (ISSRE’99), Boca Raton, FL, 1–4 Nov 1999, pp. 210

    Google Scholar 

  24. G. Fraser, A. Zeller, Mutation-driven generation of unit tests and oracles. IEEE Trans. Softw. Eng. 38(2), 278–292 (2012)

    Article  Google Scholar 

  25. T. Gibson-Robinson, P. Armstrong, A. Boulgakov, A.W. Roscoe, FDR3 - a modern refinement checker for CSP, in Tools and Algorithms for the Construction and Analysis of Systems (2014), pp. 187–201

    Google Scholar 

  26. M. Gligoric, L. Zhang, C. Pereira, G. Pokam, Selective mutation testing for concurrent code, in Proceedings of the 2013 International Symposium on Software Testing and Analysis, ISSTA 2013 (Association for Computing Machinery, New York, 2013), pp. 224–234

    Book  Google Scholar 

  27. P. Gómez-Abajo, E. Guerra, J. de Lara, A domain-specific language for model mutation and its application to the automated generation of exercises. Comput. Lang. Syst. Struct. 49, 152–173 (2017)

    Google Scholar 

  28. P. Gómez-Abajo, E. Guerra, J. de Lara, M.G. Merayo, A tool for domain-independent model mutation. Sci. Comput. Program. 163, 85–92 (2018)

    Article  Google Scholar 

  29. R. Gopinath, C. Jensen, A. Groce, Mutations: How close are they to real faults? in 25th IEEE International Symposium on Software Reliability Engineering, ISSRE 2014, Naples, 3–6 Nov 2014 (IEEE Computer Society, Washington, 2014), pp. 189–200

    Google Scholar 

  30. R. Gopinath, A. Alipour, I. Ahmed, C. Jensen, A. Groce, How hard does mutation analysis have to be, anyway? in 2015 IEEE 26th International Symposium on Software Reliability Engineering (ISSRE) (2015)

    Google Scholar 

  31. M. Harman, Y. Jia, W.B. Langdon, Strong higher order mutation-based test data generation, in Proceedings of the 19th ACM SIGSOFT symposium and the 13th European conference on Foundations of software engineering (ESEC/FSE’11) (Association for Computing Machinery, New York, 2011), pp. 212–222

    Google Scholar 

  32. M. Harman, S.A. Mansouri, Y. Zhang, Search-based software engineering: trends, techniques and applications. ACM Comput. Surv. 45(1), 11:1–11:61 (2012)

    Google Scholar 

  33. Y. Jia, M. Harman, Constructing subtle faults using higher order mutation testing, in Eighth IEEE International Working Conference on Source Code Analysis and Manipulation (SCAM 2008), Beijing, 28–29 Sept 2008, pp. 249–258

    Google Scholar 

  34. Y. Jia, M. Harman, Milu: a customizable, runtime-optimized higher order mutation testing tool for the full C language, in Proceedings of the 3rd Testing: Academic and Industrial Conference Practice and Research Techniques (TAIC PART’08), Windsor, 29–31 Aug 2008, pp. 94–98

    Google Scholar 

  35. S.-W. Kim, J.A. Clark, J.A. McDermid, Investigating the effectiveness of object-oriented testing strategies using the mutation method. Softw. Test. Verif. Reliab. 11(3), 207–225 (2001)

    Article  Google Scholar 

  36. K.N. King, A.J. Offutt, A Fortran language system for mutation-based software testing. Softw. Pract. Exper. 21(7), 685–718 (1991)

    Article  Google Scholar 

  37. M. Kintis, M. Papadakis, Y. Jia, N. Malevris, Y. Le Traon, M. Harman, Detecting trivial mutant equivalences via compiler optimisations. IEEE Trans. Softw. Eng. 44(4), 308–333 (2018)

    Article  Google Scholar 

  38. E.W. Krauser, A.P. Mathur, V. Rego, High performance software testing on SIMD machines. IEEE Trans. Softw. Eng. 17(5), 403–423 (1991)

    Article  Google Scholar 

  39. W. Krenn, R. Schlick, S. Tiran, B. Aichernig, E. Jobstl, H. Brandl, MoMut::UML model-based mutation testing for UML, in 2015 IEEE 8th International Conference on Software Testing, Verification and Validation (ICST), April 2015, pp. 1–8

    Google Scholar 

  40. M. Kusano, C. Wang, Ccmutator: a mutation generator for concurrency constructs in multithreaded C/C++ applications, in Proceedings of the 28th IEEE/ACM International Conference on Automated Software Engineering, ASE’13 (IEEE Press, Piscataway, 2013), pp. 722–725

    Google Scholar 

  41. M.Z. Kwiatkowska, G. Norman, D. Parker, PRISM 4.0: Verification of probabilistic real-time systems, in Proceedings of Computer Aided Verification - 23rd International Conference, CAV 2011, Snowbird, UT, USA, July 14–20, 2011, ed. by G. Gopalakrishnan, S. Qadeer. Lecture Notes in Computer Science, vol. 6806 (Springer, Berlin, 2011), pp. 585–591

    Google Scholar 

  42. S.C. Lee, J. Offutt, Generating test cases for XML-based web component interactions using mutation analysis, in 12th International Symposium on Software Reliability Engineering (ISSRE 2001), Hong Kong, 27–30 Nov 2001, pp. 200–209

    Google Scholar 

  43. M. Linares-Vásquez, G. Bavota, M. Tufano, K. Moran, M. Di Penta, C. Vendome, C. Bernal-Cárdenas, D. Poshyvanyk, Enabling mutation testing for Android apps, in Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering, ESEC/FSE 2017 (Association for Computing Machinery, New York, 2017), pp. 233–244

    Book  Google Scholar 

  44. T. Loise, X. Devroey, G. Perrouin, M. Papadakis, P. Heymans, Towards security-aware mutation testing, in Proceedings of the 10th International Conference on Software Testing, Verification and Validation Workshops (ICSTW’17) (2017), pp. 97–102

    Google Scholar 

  45. Y. Lou, D. Hao, L. Zhang, Mutation-based test-case prioritization in software evolution, in 2015 IEEE 26th International Symposium on Software Reliability Engineering (ISSRE) (2015), pp. 46–57

    Google Scholar 

  46. Y.-S. Ma, Y.R. Kwon, J. Offutt, Inter-class mutation operators for Java, in 13th International Symposium on Software Reliability Engineering (ISSRE 2002), Annapolis, MD, 12–15 Nov 2002, pp. 352–366

    Google Scholar 

  47. Y.-S. Ma, A.J. Offutt, Y.-R. Kwon, Mujava: a mutation system for Java, in Proceedings of the 28th international Conference on Software Engineering (ICSE ’06), Shanghai, 20–28 May 2006, pp. 827–830

    Google Scholar 

  48. L. Madeyski, W. Orzeszyna, R. Torkar, M. Józala, Overcoming the equivalent mutant problem: a systematic literature review and a comparative experiment of second order mutation. IEEE Trans. Softw. Eng. 40(1), 23–42 (2014)

    Article  Google Scholar 

  49. P.R. Mateo, M.P. Usaola, J.L.F. Alemán, Validating second-order mutation at system level. IEEE Trans. Softw. Eng. 39(4), 570–587 (2013)

    Article  Google Scholar 

  50. S. Mirshokraie, A. Mesbah, K. Pattabiraman, Guided mutation testing for JavaScript Web applications. IEEE Trans. Softw. Eng. 41(5), 429–444 (2015)

    Article  Google Scholar 

  51. A. Miyazawa, P. Ribeiro, W. Li, A. Cavalcanti, J. Timmis, J. Woodcock, RoboChart: modelling and verification of the functional behaviour of robotic applications. Softw. Syst. Model. 18(5), 3097–3149 (2019)

    Article  Google Scholar 

  52. S. Moon, Y. Kim, M. Kim, S. Yoo, Ask the mutants: mutating faulty programs for fault localization, in Verification and Validation 2014 IEEE Seventh International Conference on Software Testing (2014), pp. 153–162

    Google Scholar 

  53. T. Mouelhi, Y. Le Traon, B. Baudry, Mutation analysis for security tests qualification, in Testing: Academic and Industrial Conference Practice and Research Techniques - MUTATION (TAICPART-MUTATION 2007), Sept 2007, pp. 233–242

    Google Scholar 

  54. A.S. Namin, J.H. Andrews, D.J. Murdoch, Sufficient mutation operators for measuring test effectiveness, in Proceedings of the 30th International Conference on Software Engineering, ICSE ’08 (Association for Computing Machinery, New York, 2008), pp. 351–360

    Google Scholar 

  55. A. Offutt, The coupling effect: fact or fiction. ACM SIGSOFT Softw. Eng. Notes 14(8), 131–140 (1989)

    Article  Google Scholar 

  56. J. Offutt, Investigations of the software testing coupling effect. ACM Trans. Softw. Eng. Methodol. 1(1), 5–20 (1992)

    Article  Google Scholar 

  57. A.J. Offutt, A. Lee, G. Rothermel, R.H. Untch, C. Zapf, An experimental determination of sufficient mutant operators. ACM Trans. Softw. Eng. Methodol. 5(2), 99–118 (1996)

    Article  Google Scholar 

  58. A.J. Offutt, J. Voas, J. Payne, Mutation operators for Ada. Technical report, Information and Software Systems Engineering, George Mason University, 1996

    Google Scholar 

  59. M. Papadakis, Y. Le Traon, Using mutants to locate “Unknown” faults, in Verification and Validation 2012 IEEE Fifth International Conference on Software Testing (2012), pp. 691–700

    Google Scholar 

  60. M. Papadakis, Y. Le Traon, Metallaxis-FL: mutation-based fault localization. Softw. Test. Verif. Reliab. 25(5–7), 605–628 (2015)

    Article  Google Scholar 

  61. M. Papadakis, N. Malevris, Automatic mutation test case generation via dynamic symbolic execution, in 2010 IEEE 21st International Symposium on Software Reliability Engineering (2010), pp. 121–130

    Google Scholar 

  62. M. Papadakis, N. Malevris, Mutation based test case generation via a path selection strategy. Inf. Softw. Technol. 54(9), 915–932 (2012)

    Article  Google Scholar 

  63. A.V. Pizzoleto, F.C. Ferrari, J. Offutt, L. Fernandes, M. Ribeiro, A systematic literature review of techniques and metrics to reduce the cost of mutation testing. J. Syst. Softw. 157, 110388 (2019)

    Article  Google Scholar 

  64. M. Polo, M. Piattini, I. García-Rodríguez, Decreasing the cost of mutation testing with second-order mutants. Softw. Test. Verif. Reliab. 19(2), 111–131 (2009)

    Article  Google Scholar 

  65. R. Ramler, T. Wetzlmaier, C. Klammer, An empirical study on the application of mutation testing for a safety-critical industrial software system, in Proceedings of the Symposium on Applied Computing, SAC 2017, Marrakech, April 3–7, 2017, ed. by A. Seffah, B. Penzenstadler, C. Alves, X. Peng (ACM, New York, 2017), pp. 1401–1408

    Google Scholar 

  66. A.W. Roscoe, Understanding Concurrent Systems. Texts in Computer Science (Springer, Berlin, 2011)

    Google Scholar 

  67. G. Rothermel, R.H. Untch, C. Chu, M.J. Harrold, Prioritizing test cases for regression testing. IEEE Trans. Softw. Eng. 27(10), 929–948 (2001)

    Article  Google Scholar 

  68. D. Schuler, A. Zeller, Javalanche: efficient mutation testing for Java, in Proceedings of the 7th joint meeting of the European Software Engineering Conference and the International Symposium on Foundations of Software Engineering, Amsterdam, 24–28 Aug 2009, pp. 297–298

    Google Scholar 

  69. H. Shahriar, M. Zulkernine, Mutation-based testing of buffer overflow vulnerabilities, in Proceedings of the 32nd Annual IEEE International Computer Software and Applications Conference, COMPSAC 2008, Turku, 28 July–1 Aug 2008, pp. 979–984

    Google Scholar 

  70. D. Shin, S. Yoo, D.-H. Bae, A theoretical and empirical study of diversity-aware mutation adequacy criterion. IEEE Trans. Softw. Eng. 44(10), 914–931 (2018)

    Article  Google Scholar 

  71. D. Shin, S. Yoo, M. Papadakis, D.-H. Bae, Empirical evaluation of mutation-based test case prioritization techniques. Softw. Test. Verif. Reliab. 29(1–2), e1695 (2019)

    Google Scholar 

  72. F.C. Souza, M. Papadakis, V. Durelli, M. Delamaro, Test data generation techniques for mutation testing: a systematic mapping (2014)

    Google Scholar 

  73. T. Srivatanakul, J.A. Clark, S. Stepney, F. Polack, Challenging formal specifications by mutation: a CSP security example, in Proceedings of the 10th Asia-Pacific Software Engineering Conference (APSEC’03) (2003), pp. 340–350

    Google Scholar 

  74. M. Trakhtenbrot, New mutations for evaluation of specification and implementation levels of adequacy in testing of Statecharts models, in Proceedings of the 3rd Workshop on Mutation Analysis (MUTATION’07), Windsor, 10–14 Sept 2007, pp. 151–160

    Google Scholar 

  75. J. Tuya, M.J. Suarez Cabal, C. de la Riva, SQLmutation: a tool to generate mutants of SQL database queries, in Proceedings of the 2nd Workshop on Mutation Analysis (MUTATION’06), Raleigh, NC, Nov 2006, p. 1

    Google Scholar 

  76. J. Tuya, M.J. Suarez Cabal, C. de la Riva, Mutating database queries. Inf. Softw. Technol. 49(4), 398–417 (2007)

    Article  Google Scholar 

  77. L. Zhang, S.-S. Hou, J.-J. Hu, T. Xie, H. Mei, Is operator-based mutant selection superior to random mutant selection? in Proceedings of the 32nd ACM/IEEE International Conference on Software Engineering - Volume 1, ICSE ’10 (Association for Computing Machinery, New York, 2010), pp. 435–444

    Google Scholar 

  78. L. Zhang, M. Gligoric, D. Marinov, S. Khurshid, Operator-based and random mutant selection: better together, in 2013 28th IEEE/ACM International Conference on Automated Software Engineering (ASE) (2013), pp. 92–102

    Google Scholar 

Download references

Acknowledgements

This work has been supported by EPSRC grant EP/R025134/2 RoboTest: Systematic Model-Based Testing and Simulation of Mobile Autonomous Robots, Spanish MINECO-FEDER grant FAME RTI2018-093608-B-C31 and the Comunidad de Madrid project FORTE-CM S2018/TCS-4314.

Author information

Authors and Affiliations

  1. The University of Sheffield, Sheffield, UK

    Robert M. Hierons & Maciej Gazda

  2. Universidad Autónoma de Madrid, Madrid, Spain

    Pablo Gómez-Abajo

  3. University of Bradford, Bradford, UK

    Raluca Lefticaru

  4. Universidad Complutense de Madrid, Madrid, Spain

    Mercedes G. Merayo

Authors
  1. Robert M. Hierons
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Maciej Gazda
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Pablo Gómez-Abajo
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Raluca Lefticaru
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Mercedes G. Merayo
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Robert M. Hierons .

Editor information

Editors and Affiliations

  1. Department of Computer Science, University of York, York, UK

    Ana Cavalcanti

  2. University of Surrey, Guildford, UK

    Brijesh Dongol

  3. University of Sheffield, Sheffield, UK

    Rob Hierons

  4. University of Sunderland, Sunderland, UK

    Jon Timmis

  5. Department of Computer Science, University of York, York, UK

    Jim Woodcock

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Hierons, R.M., Gazda, M., Gómez-Abajo, P., Lefticaru, R., Merayo, M.G. (2021). Mutation Testing for RoboChart. In: Cavalcanti, A., Dongol, B., Hierons, R., Timmis, J., Woodcock, J. (eds) Software Engineering for Robotics. Springer, Cham. https://doi.org/10.1007/978-3-030-66494-7_11

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-66494-7_11

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-66493-0

  • Online ISBN: 978-3-030-66494-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation