Abstract
Traditional linear evaluation approaches are not able to address the dynamic interrelationships and feedback mechanisms involved in the increasingly complex social environment. To meet the challenges of complexity, new evaluation approaches are required. This chapter contributes to this discussion by suggesting a new integrative evaluation approach which combines foresight, multi-criteria evaluation and system dynamic modelling into the evaluation process. The developed methodology is applied in the evaluation of the Finnish Innovation Fund, Sitra.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
- 2.
See more of Sitra at https://www.sitra.fi/en/.
- 3.
Sitra law (only in Finnish) Laki Suomen itsenäisyyden juhlarahastosta. 24.8.1990/717 http://www.finlex.fi/fi/laki/ajantasa/1990/19900717
- 4.
- 5.
The more detailed results are published in the evaluation report (executive summary is available in English; the whole report is available only in Finnish) and publicly available at https://media.sitra.fi/2017/11/29120141/Selvityksia127.pdf.
- 6.
Before formulating the comprehensive system dynamic model, two narrower models were created to describe the dynamics of the two selected case areas. These models helped to create the comprehensive model.
References
Autio, E., & Thomas, L. D. W. (2013). Innovation ecosystems. In M. Dodgson, D. Gann, & N. Phillips (Eds.), Oxford handbook of innovation management (pp. 204–288). Oxford: Oxford University Press.
Berkhout, F., Smith, A., & Stirling, A. (2004). Socio-technical regimes and transition contexts. In B. Elzen, F. W. Geels, & K. Green (Eds.), System innovation and the transition to sustainability (pp. 48–75). Cheltenham: Edward Elgar. https://doi.org/10.4337/9781845423421.00013.
Cabrera, D., Colosi, L., & Lobdell, C. (2008). Systems thinking. Evaluation and Program Planning, 31, 299–310.
Castells, M. (1996). The rise of the network society, the information age: Economy, society and culture (Vol. I). Cambridge and Oxford: Blackwell.
Chen, H. T. (2005). Theory driven evaluation. In S. Mathison (Ed.), Encyclopedia of evaluation (pp. 415–419). Thousand Oaks: Sage.
Cozzens, S., & Melkers, J. (1997). Use and usefulness of performance measurement in state science and technology programs. Policy Studies Journal, 25(3), 425–435.
Denzin, N. (2006). Sociological methods: A sourcebook (5th ed.). Piscataway: Aldine Transaction.
Djellal, F., & Gallouj, F. (2013). The productivity challenge in services: Measurement and strategic perspectives. The Service Industries Journal, 33(3–4), 282–299.
Dyehouse, M., Bennett, D., Harbor, J., Childress, A., & Dark, M. (2009). A comparison of linear and systems thinking approaches for program evaluation illustrated using Indiana Interdisciplinary GK-12. Evaluation and Program Planning, 32, 187–196.
Edquist, C. (2005). Reflections on the systems of innovation approach. Science and Public Policy, 31(6), 485–489.
Elzen, B., Geels, F. W., & Green, K. (2004). System innovation and the transition to sustainability. Cheltenham and Northampton: Edward Elgar.
Eoyang, G., & Holladay, R. (2013). Adaptive action: Leveraging uncertainty in your organization. Stanford: Stanford University Press.
Forrester, J. W. (2007). System dynamics—The next fifty years. System Dynamics Review, 23(2/3), 359–370.
Forss, K., Marra, M., & Schwartz, R. (2011). Evaluating the complex: Attribution, contribution, and beyond. London: Transaction Publishers.
Freeman, C., & Louca, F. (2002). As time goes by: From the industrial revolutions to the information revolution. Oxford: Oxford University Press.
Funnel, S. C., & Rogers, P. J. (2011). Purposeful program theory: Effective use of theories of change and logic models. San Francisco: Jossey-Bass.
Geels, F. W. (2002). Technological transitions as evolutionary reconfiguration processes: A multi-level perspective and a case-study. Research Policy, 31, 1257–1274.
Geels, F. W. (2004). From sectoral systems of innovation to socio-technical systems. Insights about dynamics and change from sociology and institutional theory. Research Policy, 33, 897–920.
Geels, F. W. (2005). System innovation and the transition to sustainability. Theory, evidence and policy. Cheltenham: Edward Elgar.
Geels, F. W., & Kemp, R. (2007). Dynamics in socio-technical systems: Typology of change processes and contrasting case studies. Technology in Society, 29(4), 441–455.
Geels, F. W., & Schot, J. (2007). Typology of sociotechnical transition pathways. Research Policy, 36, 399–417.
Hansson, F. (2006). Organisational use of evaluation. Evaluation, 12(2), 159–178.
Hargreaves, M. B., & Podems, D. (2012). Advancing systems thinking in evaluation: A review of four publications. American Journal of Evaluation, 33, 462–470.
Holland, J. H. (1995). Hidden order: How adaptation builds complexity. New York: Basic Books.
Holling, C. S. (2001). Understanding the complexity of economic, ecological, and social systems. Ecosystems, 4(5), 390–405. https://doi.org/10.1007/s10021-001-0101-5.
Hyytinen, K. (2017). Supporting service innovation via evaluation: A future oriented, systemic and multi-actor approach. Doctoral dissertation 14/2017, Aalto University publication series. VTT Science 146. Retrieved from http://www.vtt.fi/inf/pdf/science/2017/S146.pdf
Hyytinen, K., Ruutu, S., Nieminen, M., Gallouj, F., & Toivonen, M. (2014). A system dynamic and multi-criteria evaluation of innovations in environmental services. Economics and Policy of Energy and the Environment, 3, 29–52.
Kellogg Foundation. (2004). Using logic models to bring together planning, evaluation & action logic model development guide. Battle Creek: Kellogg Foundation. Retrieved from https://www.wkkf.org/resource-directory/resource/2010/w-k-kellogg-foundation-evaluation-handbook.
Kemp, R., Rip, A., & Schot, J. (2001). Constructing transition paths through the management of niche. In R. Garud & P. Karnoe (Eds.), Path dependency and creation (pp. 269–299). Mahwah: Lawrence Erlbaum.
Kemp, R., & Rotmans, J. (2004). Managing the transition to sustainable mobility. In B. Elzen, F. Geels, & K. Green (Eds.), System innovation and the transition to sustainability: Theory, evidence and policy (pp. 137–167). Cheltenham: Edgar Elgar.
Kemp, R., Schot, J., & Hoogma, R. (1998). Regime shifts to sustainability through processes of niche formation. The approach of strategic niche management. Technology Analysis and Strategic Management, 10(2), 175–195.
Kivisaari, S., Lovio, R., & Väyrynen, E. (2004). Managing experiments for transition: Examples of societal embedding in energy and health care sectors. In B. Elzen, F. W. Geels, & K. Green (Eds.), System innovation and the transition to sustainability. Theory, evidence and policy (pp. 223–250). Cheltenham: Edward Elgar.
Lawrence, K. (2013). Developing leaders in VUCA environment. Retrieved January 24, 2017, from http://www.growbold.com/2013/developing-leaders-in-a-vuca-environment_UNC.2013.pdf
Martin, B. R. (2010). The origins of the concept of ‘foresight’ in science and technology: An insider’s perspective. Technological Forecasting and Social Change, 77, 1438–1447.
Martin, B. R., & Irvine, J. (1989). Research foresight: Priority-setting in science. London and New York: Pinter Publishers.
Mayne, J. (2012). Contribution analysis: Coming of age? Evaluation, 18, 270–280.
Meadows, D. H. (2008). Thinking in systems. A primer. Chelsea Green, White River Junction.
Merril, J. A., Deegan, M., Wilson, R., Kaushal, R., & Frederiks, K. (2013). A system dynamics evaluation model: Implementation of health information exchange for public health reporting. Journal of the American Medical Informatics Association, 20, 131–138.
Miles, I. (2013). Interactive impacts—Foresight as a product, service and coproduction process. In D. Dirk Meissner, L. Gokhberg, & A. Sokolov (Eds.), Technology and innovation policy for the future: Potentials and limits of foresight studies (pp. 63–82). Berlin: Springer.
Mitleton-Kelly, E. (Ed.). (2007). Complex systems and evolutionary perspectives on organizations. The application of complexity theory to organizations. Bingley, UK: Emerald.
Mowles, C. (2014). Complex, but not quite complex enough: The turn to the complexity sciences in evaluation scholarship. Evaluation, 20(2), 160–175. https://doi.org/10.1177/1356389014527885.
Nieminen, M., & Hyytinen, K. (2015). Future-oriented impact assessment: Supporting strategic decision-making in complex socio-technical environments. Evaluation, 21(4), 448–461.
Patton, M. Q. (2011). Developmental evaluation: Applying complexity concepts to enhance innovation and use. New York: Guilford.
Rip, A., & Kemp, R. (1998). Technological change. In S. Rayner & E. J. Malone (Eds.), Human choice and climate change: Vol. 2. Resources and technology (pp. 327–399). Battelle Press: Columbus.
Rip, A. (2003). Societal challenges for R&D evaluation. In P. Shapira & S. Kuhlmann (Eds.), Learning from science and technology policy evaluation: Experiences from the United States and Europe (pp. 35–59). Cheltenham: Edward Elgar.
Ritala, P., Armila, L., & Blomqvist, K.-M. (2009). Innovation orchestration capability—Defining the organizational and individual level determinants. International Journal of Innovation Management, 13(4), 569–591.
Rittel, H., & Webber Melvin, M. (1973). Dilemmas in a general theory of planning. Policy Sciences, 4, 155–169.
Rotmans, J., & Loorbach, D. (2009). Complexity and transition management. Journal of Industrial Ecology, 13(2), 184–196.
Sterman, J. D. (2001). System dynamics modeling: Tools for learning in a complex world. California Management Review, 43(4), 8–25.
Tait, J., & Williams, R. (1999). Policy approaches to research and development: foresight, framework and competitiveness. Science and Public Policy, 26(2), 101–112.
van der Knaap, P. (2006). Performance evaluation and performance management. Overcoming the downsides of policy objectives and performance indicators. Evaluation, 12(3), 278–293.
Williams, B., & Hummelbrunner, R. (2011). Systems concepts in action. A practitioner’s Toolkit. Stanford: Stanford University Press.
Williams, B., & Imam, I. (Eds.). (2007). Systems concepts in evaluation: An expert anthology. Point Reyes: Edge Press of Inverness.
Windrum, P., & GarcÃa-Goñi, M. (2008). A neo-Schumpeterian model of health services innovation. Research Policy, 37(4), 649–672.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Singapore Pte Ltd.
About this chapter
Cite this chapter
Nieminen, M., Hyytinen, K., Salminen, V., Ruutu, S. (2020). Systemic Evaluation Approach to Meet the Challenges of Complexity. In: Lehtimäki, H., Uusikylä, P., Smedlund, A. (eds) Society as an Interaction Space. Translational Systems Sciences, vol 22. Springer, Singapore. https://doi.org/10.1007/978-981-15-0069-5_5
Download citation
DOI: https://doi.org/10.1007/978-981-15-0069-5_5
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-15-0068-8
Online ISBN: 978-981-15-0069-5
eBook Packages: Economics and FinanceEconomics and Finance (R0)