Skip to main content
SpringerLink
Log in

Validation, verification, and testing techniques throughout the life cycle of a simulation study

  • Published:
Annals of Operations Research Aims and scope Submit manuscript

Abstract

Life cycle validation, verification, and testing (VV&T) is extremely important for the success of a simulation study. This paper surveys current software VV&T techniques and current simulation model VV&T techniques and describes how they can all be applied throughout the life cycle of a simulation study. The processes and credibility assessment stages of the life cycle are described and the applicability of the VV&T techniques for each stage is stated. A glossary is provided to explicitly define important terms and VV&T techniques.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. A.F. Ackerman, P.J. Fowler and R.G. Ebenau, Software inspections and the industrial production of software, in:Software Validation: Inspection, Testing, Alternatives, Alternatives, Proc. Symp. on Software Validation, Darmstadt, Germany, ed. H.-L. Hausen (1983) pp. 13–40.

  2. W.R. Adrion, M.A. Branstad and J.C. Cherniavsky, Validation, verification, and testing of computer software, Comp. Surveys 14(1982)159–192.

    Google Scholar 

  3. D.J. Aigner, A note on verification of computer simulation models, Manag. Sci. 18(1972)615–619.

    Google Scholar 

  4. F.E. Allen and J. Cocke, A program data flow analysis procedure, Commun. ACM 19(1976)137–147.

    Google Scholar 

  5. R.C. Backhouse,Program Construction and Verification (Prentice-Hall, London, 1986).

    Google Scholar 

  6. O. Balci, Requirements for model development environments, Comp. Oper. Res. 13(1986)53–67.

    Google Scholar 

  7. O. Balci, The implementation of four conceptual frameworks for simulation modeling in high-level languages, in:Proc. 1988 Winter Simulation Conf., ed. M.A. Abrams, P.L. Haigh, and J.C. Comfort (IEEE, Piscataway, NJ, 1988) pp. 287–295.

    Google Scholar 

  8. O. Balci, Guidelines for successful simulation studies, in:Proc. 1990 Winter Simulation Conf., ed. O. Balci, R.P. Sadowski, and R.E. Nance (IEEE, Piscataway, NJ, 1990) pp. 25–32.

    Google Scholar 

  9. O. Balci, Principles of simulation model validation, verification, and testing, Technical Report TR-94-24 Department of Computer Science. Virginia Tech, Blacksburg, VA (1994).

    Google Scholar 

  10. O. Balci and R.E. Nance, Formulated problem verification as an explicit requirement of model credibility, Simulation 45(1985)76–86.

    Google Scholar 

  11. O. Balci and R.E. Nance, Simulation model development environments: A research prototype, J. Oper. Res. Soc. 38(1987)753–763.

    Google Scholar 

  12. O. Balci and R.G. Sargent, A methodology for cost-risk analysis in the statistical validation of simulation models, Commun. ACM 24(1981)190–197.

    Google Scholar 

  13. O. Balci and R.G. Sargent, Some examples of simulation model validation using hypothesis testing, in:Proc. 1982 Winter Simulation Conf., ed. H.J. Highland, Y.W. Chao and O.S. Madrigal (IEEE, Piscataway, NJ, 1982) pp. 620–629.

    Google Scholar 

  14. O. Balci and R.G. Sargent, Validation of multivariate response models using Hotelling's two-sampleT 2 test, Simulation 39(1982)185–192.

    Google Scholar 

  15. O. Balci and R.G. Sargent, Validation of multivariate response trace-driven simulation models, in:Performance '83, ed. A. K. Agrawala and S.K. Tripathi (North-Holland, Amsterdam, 1983) 309–323.

    Google Scholar 

  16. O. Balci and R.G. Sargent, Validation of simulation models via simultaneous confidence intervals, Amer. J. Math. Manag. Sci. 4(1984)375–406.

    Google Scholar 

  17. J. Banks and J.S. Carson,Discrete-Event System Simulation (Prentice-Hall, Englewood Cliffs, NJ, 1984).

    Google Scholar 

  18. J. Banks, D. Gerstein and S.P. Searles, Modeling processes, validation, and verification of complex simulations: A survey, in:Methodology and Validation, ed. O. Balci (SCS, San Diego, CA, 1987) pp. 13–18.

    Google Scholar 

  19. H.P. Barendregt,The Lambda Calculus: Its Syntax and Semantics (North-Holland, New York, 1981).

    Google Scholar 

  20. T. Chusho, Test data selection and quality estimation based on the concept of essential branches for path testing, IEEE Trans. Software Eng. SE-13(1987)509–517.

    Google Scholar 

  21. K.J. Cohen and R.M. Cyert, Computer models in dynamic economics, Quarterly J. Econ. 75(1961)112–127.

    Google Scholar 

  22. M.J. Damborg and L.F. Fuller, Model validation using time and frequency domain error measures, ERDA Report 76-152, NTIS, Springfield, VA (1976).

    Google Scholar 

  23. M.S. Deutsch,Software Verification and Validation: Realistic Project Approaches (Prentice-Hall, Englewood Cliffs, NJ, 1982).

    Google Scholar 

  24. E.W. Dijkstra, Guarded commands, non-determinacy and a calculus for the derivation of programs, Commun. ACM 18(1975)453–457.

    Google Scholar 

  25. L.K. Dillon, Using symbolic execution for verification of Ada tasking programs, ACM Trans. Progr. Languages Syst. 12(1990)643–669.

    Google Scholar 

  26. J.H. Dobbins, Inspections as an up-front quality technique, in:Handbook of Software Quality Assurance, ed. G.G. Schulmeyer and J.I. McManus (Van Nostrand-Reinhold, New York, NY, 1987) pp. 137–177.

    Google Scholar 

  27. R.H. Dunn, The quest for software reliability, in:Handbook of Software Quality Assurance, ed. G.G. Schulmeyer and J.I. McManus (Van Nostrand-Reinhold, New York, NY, 1987) pp. 342–384.

    Google Scholar 

  28. S.E. Elmaghraby, The role of modeling in IE design, Ind. Eng. 19(1968)292–305.

    Google Scholar 

  29. J.R. Emshoff and R.L. Sisson,Design and Use of Computer Simulation Models (MacMillan, New York, NY, 1970).

    Google Scholar 

  30. R.E. Fairley, An experimental program-testing facility, IEEE Trans. Software Eng. SE-1(1975)350–357.

    Google Scholar 

  31. R.E. Fairley, Dynamic testing of simulation software, in:Proc. 1976 Summer Computer Simulation Conf., Washington, DC (Simulation Councils, La Jolla, CA, 1976) pp. 708–710.

    Google Scholar 

  32. G.S. Fishman,Principles of Discrete Event Simulation (Wiley-Interscience, New York, NY, 1978).

    Google Scholar 

  33. G.S. Fishman and P.J. Kiviat, The analysis of simulation generated time series, Manag. Sci. 13(1967)525–557.

    Google Scholar 

  34. J.W. Forrester,Industrial Dynamics (MIT Press, Cambridge, MA, 1961).

    Google Scholar 

  35. A.V. Gafarian and J.E. Walsh, Statistical approach for validating simulation models by comparison with operational systems, in:Proc. 4th Int. Conf. on Operations Research (Wiley, New York, NY, 1969) pp. 702–705.

    Google Scholar 

  36. A.R. Gallant, T.M. Gerig and J.W. Evans, Time series realizations obtained according to an experimental design, J. Amer. Statist. Assoc. 69(1974)639–645.

    Google Scholar 

  37. M. Garratt, Statistical validation of simulation models, in:Proc. 1974 Summer Computer Simulation Conf, Houston, TX (Simulation Councils, La Jolla, CA, 1974) pp. 915–926.

    Google Scholar 

  38. S.I. Gass, Decision-aiding models: Validation, assessment, and related issues for policy analysis, Oper. Res. 31(1983)603–631.

    Google Scholar 

  39. C.F. Hermann, Validation problems in games and simulations with special reference to models of international politics, Behav. Sci. 12(1967)216–231.

    Google Scholar 

  40. W. Hetzel,The Complete Guide to Software Testing (QED Information Sciences, Wellesley, MA, 1984).

    Google Scholar 

  41. C.P. Hollocker, The standardization of software reviews and audits, in:Handbook of Software Quality Assurance, ed. G.G. Schulmeyer and J.I. McManus (Van Nostrand-Reinhold, New York, NY, 1987) pp. 211–266.

    Google Scholar 

  42. W.E. Howden, Reliability of the path analysis testing strategy, IEEE Trans. Software Eng. SE-2(1976)208–214.

    Google Scholar 

  43. W.E. Howden, Functional program testing, IEEE Trans. Software Eng. SE-6(1980)162–169.

    Google Scholar 

  44. P. Howrey and H.H. Kelejian, Simulation versus analytical solutions, in:The Design of Computer Simulation Experiments, ed. T.H. Naylor (Duke University Press, Durham, NC, 1969) pp. 207–231.

    Google Scholar 

  45. A.W. Hunt, Statistical evaluation and verification of digital simulation models through spectral analysis, Ph.D. Dissertation, University of Texas at Austin, Austin, TX (1970).

    Google Scholar 

  46. S.H. Jacobson and E. Yücesan, On the NP-completeness of verifying structural properties of discrete event simulation models, Technical Report, Department of Industrial and Systems Engineering, Virginia Tech, Blacksburg, VA (1993).

    Google Scholar 

  47. S. Khanna, Logic programming for software verification and testing, Comp. J. 34(1991)350–357.

    Google Scholar 

  48. N.A. Kheir and W.M. Holmes, On validating simulation models of missile systems, Simulation 30(1978)117–128.

    Google Scholar 

  49. J.C. King, Symbolic execution and program testing, Commun. ACM 19(1976)385–394.

    Google Scholar 

  50. J.P.C. Kleijnen,Statistical Techniques in Simulation, Vol. 2 (Marcel Dekker, New York, NY, 1975).

    Google Scholar 

  51. P.L. Knepell and D.C. Arangno,Simulation Validation: A Confidence Assessment Methodology, Monograph 3512-04 (IEEE Computer Society Press, Los Alamitos, CA, 1993).

    Google Scholar 

  52. J.C. Knight and E.A. Myers, An improved inspection technique, Commun. ACM 36(1993)51–61.

    Google Scholar 

  53. A.M. Law, Statistical analysis of simulation output data, Oper. Res. 31(1983)983–1029.

    Google Scholar 

  54. A.M. Law and W.D. Kelton,Simulation Modeling and Analysis, 2nd ed. (McGraw-Hill, New York, NY, 1991).

    Google Scholar 

  55. Z. Manna, S. Ness and J. Vuillemin, Inductive methods for proving properties of programs, Commun. ACM 16(1973)491–502.

    Google Scholar 

  56. J. Martin and C. McClure,Diagramming Techniques for Analysts and Programmers (Prentice-Hall, Englewood Cliffs, NJ, 1985).

    Google Scholar 

  57. D.K. Miller, Validation of computer simulations in the social sciences, in:Proc. 6th Annual Conf. on Modeling and Simulation (Pittsburg, PA, 1975) pp. 743–746.

  58. D.R. Miller, Model validation through sensitivity analysis, in:Proc. 1974 Summer Computer Simulation Conf., Houston, TX (Simulation Councils, La Jolla, CA, 1974) pp. 911–914.

    Google Scholar 

  59. D.R. Miller, Sensitivity analysis and validation of simulation models, J. Theor. Biol. 48(1974)345–360.

    Google Scholar 

  60. R.L. Moose and R.E. Nance, The design and development of an analyzer for discrete event model specifications, in:Impacts of Recent Computer Advances on Operations Research, ed. R. Sharda, B.L. Golden, E. Wasil, O. Balci and W. Stewart (Elsevier, New York, NY, 1989) pp. 407–421.

    Google Scholar 

  61. G.J. Myers, A controlled experiment in program testing and code walkthroughs/inspections, Commun. ACM 21(1978)760–768.

    Google Scholar 

  62. G.J. Myers,The Art of Software Testing (Wiley, New York, NY, 1979).

    Google Scholar 

  63. R.E. Nance, The feasibility of and methodology for developing federal documentation standards for simulation models: Final report to the National Bureau of Standards, Department of Computer Science, VPI&SU, Blacksburg, VA (1977).

    Google Scholar 

  64. R.E. Nance, Model representation in discrete event simulation: The conical methodology, Technical Report CS81003-R, Department of Computer Science, VPI&SU, Blacksburg, VA (1981).

    Google Scholar 

  65. R.E. Nance, The conical methodology: A framework for simulation model development, in:Methodology and Validation, ed. O. Balci, (SCS, San Diego, CA, 1987) pp. 38–43.

    Google Scholar 

  66. R.E. Nance, Conical methodology: An evolutionary convergence of systems and software engineering, Ann. Oper. Res. 53(1994), this volume.

  67. R.E. Nance and C.M. Overstreet, Diagnostic assistance using digraph representations of discrete event simulation model specifications. Trans. SCS 4(1987)33–57.

    Google Scholar 

  68. T.H. Naylor and J.M. Finger, Verification of computer simulation models, Manag. Sci. 14(1967)B92-B101.

    Google Scholar 

  69. M.A. Ould and C. Unwin,Testing in Software Development (Cambridge University Press, Cambridge, 1986).

    Google Scholar 

  70. C.M. Overstreet and R.E. Nance, A specification language to assist in analysis of discrete event simulation models, Commun. ACM 28(1985)190–201.

    Google Scholar 

  71. T.I. Ören, Concepts and criteria to assess acceptability of simulation studies: A frame of reference, Commun. ACM 24(1981)180–189.

    Google Scholar 

  72. T.I. Ören, Artificial intelligence in quality assurance of simulation studies, in:Modelling and Simulation Methodology in the Artificial Intelligence Era, ed. M.S. Elzas, T.I. Ören and B.P. Zeigler (North Holland, Amsterdam, 1986) pp. 267–278.

    Google Scholar 

  73. T.I. Ören, Quality assurance paradigms for artificial intelligence in modelling and simulation, Simulation 48(1987)149–151.

    Google Scholar 

  74. R.J. Paul, Visual simulation: Seeing is believing?, in:Impaçts of Recent ComputerAdvances on Operations Research, ed. R. Sharda, B.L. Golden, E. Wasil, O. Balci, and W. Stewart (Elsevier, New York, NY, 1989) pp. 422–432.

    Google Scholar 

  75. R.E. Prather and J.P. Myers, Jr., The path prefix software testing strategy, IEEE Trans. Software Eng. SE-13(1987)761–766.

    Google Scholar 

  76. C.V. Ramamoorthy, S.F. Ho and W.T. Chen, On the automated generation of program test data, IEEE Trans. Software Eng. SE-2(1976)293–300.

    Google Scholar 

  77. C. Reynolds and R.T. Yeh, Induction as the basis for program verification, IEEE Trans. Software Eng. SE-2(1976)244–252.

    Google Scholar 

  78. D.J. Richardson and L.A. Clarke, Partition analysis: A method combining testing and verification, IEEE Trans. Software Eng. SE-11(1985)1477–1490.

    Google Scholar 

  79. J.R. Rowland and W.M. Holmes, Simulation validation with sparse random data, Comp. Elect. Eng. 5(1978)37–49.

    Google Scholar 

  80. R.G. Sargent, Validation and verification of simulation models, in:Proc. 1992 Winter Simulation Conf., ed. J.J. Swain, D. Goldsman, R.C. Crain, and J.R. Wilson (IEEE, Piscataway, NJ, 1992) pp. 104–114.

    Google Scholar 

  81. E. Satterthwaite, Debugging tools for high level languages, Software — Practice and Experience 2(1972)197–217.

    Google Scholar 

  82. S.R. Schach,Software Engineering, 2nd ed. (Irwin, Homewood, IL 1993).

    Google Scholar 

  83. S. Schlesinger et al., Terminology for model credibility, Simulation 32(1979)103–104.

    Google Scholar 

  84. B. Schmeiser, Random variate generation, in:Proc. 1981 Winter Simulation Conf., ed. T.I. Ören, C.M. Delfosse and C.M. Shub (IEEE, Piscataway, NJ, 1981) pp. 227–242.

    Google Scholar 

  85. T.J. Schriber,Simulation Using GPSS (Wiley, New York, NY, 1974).

    Google Scholar 

  86. L.W. Schruben, Establishing the credibility of simulations, Simulation 34(1980)101–105.

    Google Scholar 

  87. R.E. Shannon,Systems Simulation: The Art and Science (Prentice-Hall, Englewood Cliffs, NJ, 1975).

    Google Scholar 

  88. L.G. Stucki, New directions in automated tools for improving software quality, in:Current Trends in Programming Methodology, Vol. 2, ed. R. Yeh (Prentice-Hall, Englewood Cliffs, NJ, 1977) pp. 80–111.

    Google Scholar 

  89. T.J. Teorey, Validation criteria for computer system simulations, Simuletter 6(1975)9–20.

    Google Scholar 

  90. H. Theil,Economic Forecasts and Policy (North-Holland, Amsterdam, 1961).

    Google Scholar 

  91. A.M. Turing, Computing machinery and intelligence, in:Computers and Thought, ed. E.A. Feigenbaum and J. Feldman (McGraw-Hill, New York, NY, 1963) pp. 11–15.

    Google Scholar 

  92. T.P. Tytula, A method for validating missile system simulation models, Technical Report E-78-11, U.S. Army Missile R&D Command, Redstone Arsenal, AL (1978).

  93. U.S. GAO,DOD Simulations: Improved Assessment Procedures Would Increase the Credibility of Results, U.S. General Accounting Office GAO/PEMD-88-3, Washington, DC (1987).

  94. R.L. Van Horn, Validation of simulation results, Manag. Sci. 17(1971)247–258.

    Google Scholar 

  95. D. Watts, Time series analysis, in:The Design of Computer Simulation Experiments, ed. T.H. Naylor (Duke University Press, Durham, NC, 1969) pp. 165–179.

    Google Scholar 

  96. R.B. Whitner and O. Balci, Guidelines for selecting and using simulation model verification techniques, in:Proc. 1989 Winter Simulation Conf., ed. E.A. MacNair, K.J. Musselman and P. Heidelberger (IEEE, Piscataway, NJ, 1989) pp. 559–568.

    Google Scholar 

  97. J.R. Wilson and A.A.B. Pritsker, A survey of research on the simulation startup problem, Simulation 31(1978)55–58.

    Google Scholar 

  98. R.N. Woolley and M. Pidd, Problem structuring — A literature review, J. Oper. Res. Soc. 32(1981)197–206.

    Google Scholar 

  99. R.D. Wright, Validating dynamic models: An evaluation of tests of predictive power, in:Proc. 1972 Summer Computer Simulation Conf., San Diego, CA, (Simulation Councils, La Jolla, CA, 1972) pp. 1286–1296.

    Google Scholar 

  100. R.T. Yeh, Verification of programs by predicate transformation, in:Current Trends in Programming Methodology, Vol. 2, ed. R. Yeh (Prentice-Hall, Englewood Cliffs, NJ, 1977) pp. 228–247.

    Google Scholar 

  101. E. Yourdon,Structured Walkthroughs, 3rd ed. (Yourdon Press, New York, NY, 1985).

    Google Scholar 

  102. E. Yücesan and S.H. Jacobson, Building correct simulation models is difficult, in:Proc. 1992 Winter Simulation Conf., ed. J.J. Swain, D. Goldsman, R.C. Crain, and J.R. Wilson (IEEE, Piscataway, NJ, 1992) pp. 783–790.

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Computer Science, Virginia Polytechnic Institute and State University, 24061-0106, Blacksburg, VA, USA

    Osman Balci

Authors
  1. Osman Balci
    View author publications

    You can also search for this author in PubMed Google Scholar

Rights and permissions

Reprints and permissions

About this article

Cite this article

Balci, O. Validation, verification, and testing techniques throughout the life cycle of a simulation study. Ann Oper Res 53, 121–173 (1994). https://doi.org/10.1007/BF02136828

Download citation

  • Issue Date:

  • DOI: https://doi.org/10.1007/BF02136828

Keywords

Search

Navigation