Abstract
Over the past thirty years, the visualization community has developed theories and models to explain visualization as a technology that augments human cognition by enabling the efficient, accurate, and timely discovery of meaningful information in data. Along the way, practitioners have also debated theories and practices for visualization evaluation: How do we generate durable, reliable evidence that a visualization is effective? Interestingly, there is still no consensus in the visualization research community how to evaluate visualization methods. The goal of this chapter is to rise awareness of still open issues in the visualization evaluation and to discuss appropriate evaluations suitable for different visualization approaches. This includes user studies and best practices to conduct them but also other approaches for suitable evaluation of visualization. The chapter is structured as a moderated dialog of two visualization experts.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Abdul-Rahman, A., Chen, M., Laidlaw, D.H.: A survey of variables used in empirical studies for visualization. In: Chen, M., Hauser, H., Rheingans, P., Scheuermann, G. (eds.) Foundations of Data Visualization. Springer, Berlin (2019)
Chen, M., Ebert, D.S.: An ontological framework for supporting the design and evaluation of visual analytics systems. Comput. Graph. Forum 38(3), 131–144. https://onlinelibrary.wiley.com/doi/abs/10.1111/cgf.13677 (2019). https://doi.org/10.1111/cgf.13677
Demiralp, C., Jackson, C., Karelitz, D., Zhang, S., Laidlaw, D.H.: Cave and fishtank virtual-reality displays: a qualitative and quantitative comparison. IEEE Trans. Vis. Comput. Graph. 12(3), 323–330 (2006)
Gillmann, C., Wischgoll, T., Hamann, B., Hagen, H.: Accurate and reliable extraction of surfaces from image data using a multi-dimensional uncertainty model. Graph. Model. 99, 13–21. http://www.sciencedirect.com/science/article/pii/S1524070318300365 (2018). https://doi.org/10.1016/j.gmod.2018.07.004
Glendenning, K., Wischgoll, T., Harris, J., Vickery, R., Blaha, L.: Parameter space visualization for large-scale datasets using parallel coordinate plots. J. Imaging Sci. Technol. 60(1), 10,406–1–10,406–8 (2016)
Gomez, S.R., Guo, H., Ziemkiewicz, C., Laidlaw, D.H.: An insight- and task-based methodology for evaluating spatiotemporal visual analytics. In: Proceedings of IEEE VAST (2014)
Guo, H., Gomez, S.R., Ziemkiewicz, C., Laidlaw, D.H.: A case study using visualization interaction logs and insight metrics to understand how analysts arrive at insights. In: Proceedings of IEEE VAST (2015)
Holten, D., Van Wijk, J.J.: Force-directed edge bundling for graph visualization. Comput. Graph. Forum 28(3), 983–990 (2009). https://doi.org/10.1111/j.1467-8659.2009.01450.x
Isenberg, T., Isenberg, P., Chen, J., Sedlmair, M., Möller, T.: A systematic review on the practice of evaluating visualization. IEEE Trans. Vis. Comput. Graph. 19(12), 2818–2827 (2013). https://doi.org/10.1109/TVCG.2013.126
Keefe, D., Acevedo, D., Moscovich, T., Laidlaw, D.H., LaViola, J.: Cavepainting: a fully immersive 3D artistic medium and interactive experience. In: Proceedings of ACM Symposium on Interactive 3D Graphics, pp. 85–93 (2001)
Kirby, M., Marmanis, H., Laidlaw, D.H.: Visualizing multivalued data from 2D incompressible flows using concepts from painting. Proc. IEEE Vis. 1999, 333–340 (1999)
Koehler, C., Wischgoll, T., Dong, H., Gaston, Z.: Vortex visualization in ultra low reynolds number insect flight. IEEE Trans. Vis. Comput. Graph. 17(12), 2071–2079 (2011). https://doi.org/10.1109/TVCG.2011.260
Kosara, R., Healey, C.G., Interrante, V., Laidlaw, D.H., Ware, C.: User studies: why, how, and when. Comput. Graph. Appl. 23(4), 20–25 (2003)
Laidlaw, D.H., Kirby, M., Jackson, C., Davidson, J.S., Miller, T., DaSilva, M., Warren, W., Tarr, M.: Comparing 2D vector field visualization methods: a user study. IEEE Trans. Vis. Comput. Graph. 11(1), 59–70 (2005)
Lam, H., Bertini, E., Isenberg, P., Plaisant, C., Carpendale, S.: Empirical studies in information visualization: seven scenarios. IEEE Trans. Vis. Comput. Graph. 18(9), 1520–1536 (2012). https://doi.org/10.1109/TVCG.2011.279
Lam, H., Tory, M., Munzner, T.: Bridging from goals to tasks with design study analysis reports. IEEE Trans. Vis. Comput. Graph. 24(1), 435–445 (2018). https://doi.org/10.1109/TVCG.2017.2744319
Lorensen, W.E., Cline, H.E.: Marching cubes: a high resolution 3D surface construction algorithm. SIGGRAPH Comput. Graph. 21(4), 163–169 (1987). https://doi.org/10.1145/37402.37422
Matkovic, K., Gracanin, D., Jelovic, M., Hauser, H.: Interactive visual steering - rapid visual prototyping of a common rail injection system. IEEE Trans. Vis. Comput. Graph. 14(6), 1699–1706 (2008). https://doi.org/10.1109/TVCG.2008.145
Meyer, M., Sedlmair, M., Munzner, T.: The four-level nested model revisited: blocks and guidelines. In: Proceedings of the 2012 BELIV Workshop: Beyond Time and Errors - Novel Evaluation Methods for Visualization, BELIV ’12, pp. 11:1–11:6. ACM, New York, NY, USA (2012). https://doi.org/10.1145/2442576.2442587
Munzner, T.: A nested model for visualization design and validation. IEEE Trans. Vis. Comput. Graph. 15(6), 921–928 (2009). https://doi.org/10.1109/TVCG.2009.111
Sedlmair, M., Meyer, M., Munzner, T.: Design study methodology: reflections from the trenches and the stacks. IEEE Trans. Vis. Comput. Graph. 18(12), 2431–2440 (2012). https://doi.org/10.1109/TVCG.2012.213
Smith, G.C.S., Pell, J.P.: Parachute use to prevent death and major trauma related to gravitational challenge: systematic review of randomised controlled trials. BMJ 327(7429), 1459–1461. https://www.bmj.com/content/327/7429/1459 (2003). https://doi.org/10.1136/bmj.327.7429.1459
Upson, C., Faulhaber, T.A., Kamins, D., Laidlaw, D., Schlegel, D., Vroom, J., Gurwitz, R., van Dam, A.: The application visualization system: a computational environment for scientific visualization. IEEE Comput. Graph. Appl. 9(4), 30–42 (1989). https://doi.org/10.1109/38.31462
van Dam, A., Laidlaw, D.H., Simpson, R.M.: Experiments in immersive virtual reality for scientific visualization. Comput. Graph. 26(4), 535–555 (2002)
Weber, G.H., Carpendale, S., Ebert, D., Fisher, B., Hagen, H., Shneiderman, B., Ynnerman, A.: Apply or die: on the role and assessment of application papers in visualization. IEEE Comput. Graph. Appl. 37(3), 96–104 (2017). https://doi.org/10.1109/MCG.2017.51
Wischgoll, T., Glines, M., Whitlock, T., Guthrie, B.R., Mowrey, C.M., Parikh, P.J., Flach, J.: Display infrastructure for virtual environments (dive). J. Imaging Sci. Technol. 61(6), 60,406–1–60,406–11 (2017)
Wischgoll, T., Scheuermann, G.: Detection and visualization of closed streamlines in planar flows. IEEE Trans. Vis. Comput. Graph. 7(2), 165–172 (2001). https://doi.org/10.1109/2945.928168
Ziemkiewicz, C., Chen, M., Laidlaw, D., Preim, B., Weiskopf, D.: Open challenges in empirical visualization research. In: Chen, M., Hauser, H., Rheingans, P., Scheuermann, G. (eds.) Foundations of Data Visualization. Springer, Berlin (2019)
Acknowledgements
The authors would like to thank Laura McNamara for numerous discussions. Her input was very valuable and it helped improve the chapter considerably.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this chapter
Cite this chapter
Matković, K., Wischgoll, T., Laidlaw, D.H. (2020). Empirical Evaluations with Domain Experts. In: Chen, M., Hauser, H., Rheingans, P., Scheuermann, G. (eds) Foundations of Data Visualization. Springer, Cham. https://doi.org/10.1007/978-3-030-34444-3_8
Download citation
DOI: https://doi.org/10.1007/978-3-030-34444-3_8
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-34443-6
Online ISBN: 978-3-030-34444-3
eBook Packages: Computer ScienceComputer Science (R0)