Abstract
In this paper, we show how experiments on the understandability of business process models can depend on the exact wording used in the experiments’ questionnaires. For this purpose, we partially replicated a published experiment. We asked a group of students a number of questions on relations between tasks in a business process model. Alternatively, we used a set of modified questions which were aimed to ask for exactly the same relations. The result was that there was a significant difference in the number of correct answers between the two systems to construct a question. We argue that a non-negligible part of the wrong answers given in the experiment did not result from problems to understand the model, but rather from problems to understand the question. It follows that it is dangerous to draw conclusions from such an experiment until enough effort has been taken to select appropriate questions.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Lindland, O.I., Sindre, G., Sølvberg, A.: Understanding quality in conceptual modeling. IEEE Softw. 11, 42–49 (1994)
Becker, J., Rosemann, M., Uthmann, C.v.: Guidelines of business process modeling. In: Business Process Management, Models, Techniques, and Empirical Studies, pp. 30–49. Springer, London (2000)
Rittgen, P.: Quality and perceived usefulness of process models. In: SAC 2010: Proceedings of the 2010 ACM Symposium on Applied Computing, pp. 65–72. ACM, New York (2010)
Schuman, H., Presser, S.: Questions and Answers in Attitude Surveys. Academic Press, San Diego (1981)
Carmines, E., Zeller, R.: Reliability and Validity Assessment. Sage Univ. papers, Thousand Oaks (1979)
Cardoso, J.: Process control-flow complexity metric: An empirical validation. In: IEEE International Conference on Services Computing, pp. 167–173 (2006)
Holschke, O., Rake, J., Levina, O.: Granularity as a cognitive factor in the effectiveness of business process model reuse. In: Dayal, U., Eder, J., Koehler, J., Reijers, H.A. (eds.) BPM 2009. LNCS, vol. 5701, pp. 245–260. Springer, Heidelberg (2009)
Aguilar, E.R., Sanchez, L., Carballeira, F.G., Ruiz, F., Piattini, M., Caivano, D., Visaggio, G.: Prediction models for BPMN usability and maintainability. In: 2009 IEEE Conference on Commerce and Enterprise Computing, pp. 383–390 (2009)
Sarshar, K., Loos, P.: Comparing the control-flow of EPC and petri net from the end-user perspective. In: van der Aalst, W.M.P., Benatallah, B., Casati, F., Curbera, F. (eds.) BPM 2005. LNCS, vol. 3649, pp. 434–439. Springer, Heidelberg (2005)
Canfora, G., GarcĂa, F., Piattini, M., Ruiz, F., Visaggio, C.A.: A family of experiments to validate metrics for software process models. J. Syst. Softw. 77, 113–129 (2005)
Lara Proano, M.D.: Visual layout for drawing understandable process models. Master’s thesis, Technische Universiteit Eindhoven (2008)
Reijers, H., Mendling, J.: Modularity in process models: Review and effects. In: Dumas, M., Reichert, M., Shan, M.-C. (eds.) BPM 2008. LNCS, vol. 5240, pp. 20–35. Springer, Heidelberg (2008)
Mendling, J., Reijers, H.A., Cardoso, J.: What makes process models understandable? In: Alonso, G., Dadam, P., Rosemann, M. (eds.) BPM 2007. LNCS, vol. 4714, pp. 48–63. Springer, Heidelberg (2007)
Recker, J., Dreiling, A.: Does it matter which process modelling language we teach or use? In: 18th Australasian Conference on Information Systems (2007)
Mendling, J., Strembeck, M.: Influence factors of understanding business process models. In: 11th International Conference, Business Information Systems, BIS 2008, Innsbruck, Austria, pp. 142–153. Springer, Heidelberg (2008)
Vanderfeesten, I.T.P., Reijers, H.A., Mendling, J., van der Aalst, W.M.P., Cardoso, J.: On a quest for good process models: The cross-connectivity metric. In: Bellahsène, Z., Léonard, M. (eds.) CAiSE 2008. LNCS, vol. 5074, pp. 480–494. Springer, Heidelberg (2008)
Mayer, R.: Models for understanding. Rev. of Educational Research 59, 43 (1989)
Melcher, J., Seese, D.: Process measurement: Insights from software measurement on measuring process complexity, quality and performance. Technical report, Universität Karlsruhe, TH (2008)
Melcher, J., Seese, D.: Towards validating prediction systems for process understandability: Measuring process understandability. In: 10th International Symposium on Symbolic and Numeric Algorithms for Scientific Computing, pp. 564–571. IEEE Computer Society, Los Alamitos (2008)
Melcher, J., Mendling, J., Reijers, H.A., Seese, D.: On measuring the understandability of process models. In: Revised Papers of the BPM 2009 International Workshops. LNBIP, vol. 43, pp. 465–476. Springer, Ulm (2010)
Dwyer, M.B., Avrunin, G.S., Corbett, J.C.: Patterns in property specifications for finite-state verification. In: Proc. of the 21st International Conference on Software Engineering, pp. 411–420. IEEE Computer Society Press, Los Alamitos (1999)
Sudman, S., Bradburn, N.M.: Asking Questions: A Practical Guide to Questionnaire Design. Jossey-Bass, San Francisco (1982)
Converse, J.M., Presser, S.: Survey Questions: Handcrafting the Standardized Questionnaire. Sage Publications, Thousand Oaks (1986)
Aranda, J., Ernst, N., Horkoff, J., Easterbrook, S.M.: A framework for empirical evaluation of model comprehensibility. In: International Workshop on Modeling in Software Engineering, MiSE 2007 (2007)
Patig, S.: A practical guide to testing the understandability of notations. In: Fifth Asia-Pacific Conference on Conceptual Modelling (APCCM 2008). CRPIT, vol. 79, pp. 49–58. Australian Computer Society (2008)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2011 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Laue, R., Gadatsch, A. (2011). Measuring the Understandability of Business Process Models - Are We Asking the Right Questions?. In: zur Muehlen, M., Su, J. (eds) Business Process Management Workshops. BPM 2010. Lecture Notes in Business Information Processing, vol 66. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-20511-8_4
Download citation
DOI: https://doi.org/10.1007/978-3-642-20511-8_4
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-20510-1
Online ISBN: 978-3-642-20511-8
eBook Packages: Computer ScienceComputer Science (R0)