Skip to main content
SpringerLink
Log in

Towards a Transdisciplinary Evaluation Framework for Mobile Cross-Border Government Services

  • Conference paper
  • First Online:
Electronic Government (EGOV 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13391))

Included in the following conference series:

  • 1245 Accesses

Abstract

The evaluation and assessment of project results and their impact are still a recurring challenge in the digital government discipline. Many technologically driven projects or products have faced challenges, where the technology is advanced, but the market adoption and user acceptance are still lacking. To counter these challenges, this paper presents a transdisciplinary evaluation framework and how it could be applied. The foundation for the evaluation framework was a literature review on the most recent and relevant academic publications on transdisciplinary evaluations, which was narrowed down by using selected relevant search terms. This theoretical background was enhanced by a series of practical workshops to validate the findings. By using a transdisciplinary approach, this paper presents a transdisciplinary evaluation framework that enhances the evaluation process of project results in the digital government discipline with six pillars to reflect (1) the real word context, (2) interdisciplinary research, (3) going beyond science, (4) interaction (5) integration, and (6) relevance. Alongside these pillars, dimensions of measurement for the evaluation are also presented and elaborated on. While this evaluation framework could be adopted for many types of projects or products, this paper showcases how it is applied for an international digital government pilot research project throughout its development process. It presents the methodology and process used in establishing the evaluation framework, the evaluation framework itself, and a short discussion.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Ndaguba, E.A., Ijeoma, E.O.C.: Exploring the epistemology of transdisciplinarity in public policy and administration in South Africa. TDSA 13, 1–13 (2017). https://doi.org/10.4102/td.v13i1.406

    Article  Google Scholar 

  2. Hans, H.J.: Discipline or interdisciplinary study domain? Challenges and promises in electronic government research. In: Chen, H., et al. (eds.) Digital Government, pp. 21–41. Springer US, Boston, MA (2008). https://doi.org/10.1007/978-0-387-71611-4_2

    Chapter  Google Scholar 

  3. Heeks, R., Bailur, S.: Analyzing e-government research: perspectives, philosophies, theories, methods, and practice. Gov. Inf. Q. 24, 243–265 (2007). https://doi.org/10.1016/j.giq.2006.06.005

    Article  Google Scholar 

  4. Fedorowicz, J., Dias, M.A.: A decade of design in digital government research. Gov. Inf. Q. 27, 1–8 (2010). https://doi.org/10.1016/j.giq.2009.09.002

    Article  Google Scholar 

  5. Gil-García, J.R., Luna-Reyes, L.F.: Integrating conceptual approaches to e-government. In: Encyclopedia of E-commerce, E-government, and Mobile Commerce, pp. 636–643. IGI Global (2006)

    Google Scholar 

  6. OECD, International Telecommunication Union: M-Government: Mobile Technologies for Responsive Governments and Connected Societies. OECD (2011). https://doi.org/10.1787/9789264118706-en

  7. Höchtl, B., Lampoltshammer, T.J.: Rechtliche Rahmenbedingungen und technische Umsetzung von E-Government in Österreich. In: Stember, J., Eixelsberger, W., Spichiger, A., Neuroni, A., Habbel, F.-R., Wundara, M. (eds.) Handbuch E-Government, pp. 135–161. Springer, Wiesbaden (2019). https://doi.org/10.1007/978-3-658-21402-9_10

    Chapter  Google Scholar 

  8. Esteves, J., Joseph, R.C.: A comprehensive framework for the assessment of eGovernment projects. Gov. Inf. Q. 25, 118–132 (2008). https://doi.org/10.1016/j.giq.2007.04.009

    Article  Google Scholar 

  9. Qureshi, H.A., Salman, Y., Irfan, S., Jabeen, N.: A systematic review of e-government evaluation. Pak. Econ. Soc. Rev. 37 (2017)

    Google Scholar 

  10. Liu, J., Derzsi, Z., Raus, M., Kipp, A.: eGovernment project evaluation: an integrated framework. In: Wimmer, M.A., Scholl, H.J., Ferro, E. (eds.) EGOV 2008. LNCS, vol. 5184, pp. 85–97. Springer, Heidelberg (2008). https://doi.org/10.1007/978-3-540-85204-9_8

    Chapter  Google Scholar 

  11. Sellung, R., Roßnagel, H.: Evaluating complex identity management systems – the futureid approach, 7 (2015)

    Google Scholar 

  12. Walter, A.I., Helgenberger, S., Wiek, A., Scholz, R.W.: Measuring societal effects of transdisciplinary research projects: design and application of an evaluation method. Eval. Program Plann. 30, 325–338 (2007). https://doi.org/10.1016/j.evalprogplan.2007.08.002

    Article  Google Scholar 

  13. Wolf, B., Lindenthal, T., Szerencsits, M., Holbrook, J.B., Heß, J.: Evaluating research beyond scientific impacthow to include criteria for productive interactions and impact on practice and society. GAIA-Ecol. Perspect. Sci. Soc. 22, 104–114 (2013)

    Google Scholar 

  14. Klein, J.T.: Evaluation of interdisciplinary and transdisciplinary research. Am. J. Prev. Med. 35, S116–S123 (2008). https://doi.org/10.1016/j.amepre.2008.05.010

    Article  Google Scholar 

  15. Binder, C.R., Absenger-Helmli, I., Schilling, T.: The reality of transdisciplinarity: a framework-based self-reflection from science and practice leaders. Sustain. Sci. 10(4), 545–562 (2015). https://doi.org/10.1007/s11625-015-0328-2

    Article  Google Scholar 

  16. Zscheischler, J., Rogga, S., Weith, T.: Experiences with transdisciplinary research: sustainable land management third year status conference. Syst. Res. Behav. Sci. 31, 751–756 (2014)

    Article  Google Scholar 

  17. Grimsley, M., Meehan, A., Tan, A.: Evaluative design of e-government projects: a community development perspective. Transforming Gov.: People Process. Policy 1, 174–193 (2007). https://doi.org/10.1108/17506160710751995

    Article  Google Scholar 

  18. Venable, J., Pries-Heje, J., Baskerville, R.: A comprehensive framework for evaluation in design science research. In: Peffers, K., Rothenberger, M., Kuechler, B. (eds.) DESRIST 2012. LNCS, vol. 7286, pp. 423–438. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-29863-9_31

    Chapter  Google Scholar 

  19. European Commission: Communication of 8 December 1999 on a Commission initiative for the special European Council of Lisbon, 23 and 24 March 2000 - eEurope - An information society for all (1999)

    Google Scholar 

  20. European Commission: Communication of 28 May 2002 from the Commission to the Council, the European Parliament, the Economic and Social Committee and the Committee of the Regions - The eEurope 2005 action plan: an information society for everyone (COM(2002) 263 final) (2002)

    Google Scholar 

  21. European Union: Malmö Declaration - Ministerial Declaration on eGovernment (2009)

    Google Scholar 

  22. Krimmer, R., Webster, W.: Trust, security and public services in the digital age. Forthcoming (2021)

    Google Scholar 

  23. European Commission: Regulation (EU) 910/2014 of the European Parliament and of the Council of 23 July 2014 on electronic identification and trust services for electronic transactions in the internal market and repealing Directive 1999/93/EC. 2014 (2014)

    Google Scholar 

  24. European Union: Tallinn Declaration on eGovernment. 2017. p. at the ministerial meeting during Estonian Presidency of the Council of the EU on 6 October 2017 (2017)

    Google Scholar 

  25. Schmidt, C., Krimmer, R.: How to implement the European digital single market: identifying the catalyst for digital transformation. J. Eur. Integr. 44, 59–80 (2022). https://doi.org/10.1080/07036337.2021.2011267

    Article  Google Scholar 

  26. European Commission: Large Scale Pilots (2016). https://cordis.europa.eu/programme/id/H2020_IoT-01-2016. Accessed 16 March 2021

  27. European Union: Regulation (EU) 2018/1724 of the European Parliament and of the Council of 2 October 2018 establishing a single digital gateway to provide access to information, to procedures and to assistance and problem-solving services and amending Regulation (EU) No 1024/2012 (2018)

    Google Scholar 

  28. Krimmer, R., Kalvet, T., Toots, M., Cepilovs, A., Tambouris, E.: Exploring and demonstrating the once-only principle. In: 18th Annual International Conference on Digital Government, pp. 546–551 (2017). https://doi.org/10.1145/3085228.3085235

  29. Schmidt, C., Krimmer, R., Lampoltshammer, T.J.: “When need becomes necessity” - the single digital gateway regulation and the once-only principle from a European point of view, p. 6 (2021)

    Google Scholar 

  30. Krimmer, R., Prentza, A., Mamrot, S., Schmidt, C., Cepilovs, A.: The future of the once-only principle in Europe. In: Krimmer, R., Prentza, A., Mamrot, S. (eds.) The Once-Only Principle. LNCS, vol. 12621, pp. 225–236. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-79851-2_12

    Chapter  Google Scholar 

  31. Collatto, D.C., Dresch, A., Lacerda, D.P., Bentz, I.G.: Is action design research indeed necessary? Analysis and synergies between action research and design science research. Syst. Pract. Action Res. 31(3), 239–267 (2017). https://doi.org/10.1007/s11213-017-9424-9

    Article  Google Scholar 

  32. Klock, A.C.T., Gasparini, I., Pimenta, M.S.: 5W2H Framework: a guide to design, develop and evaluate the user-centered gamification. In: Proceedings of the 15th Brazilian Symposium on Human Factors in Computing Systems, pp. 1–10 (2016)

    Google Scholar 

  33. Piggot-Irvine, E., Zornes, D.: Developing a framework for research evaluation in complex contexts such as action research. Sage Open 6, 215824401666380 (2016). https://doi.org/10.1177/2158244016663800

    Article  Google Scholar 

  34. Pregernig, M.: Transdisciplinarity viewed from afar: science-policy assessments as forums for the creation of transdisciplinary knowledge. Sci. Public Policy 33, 445–455 (2006)

    Article  Google Scholar 

  35. Heilmann, A., Reinhold, S.: Evaluation of a transdisciplinary research project for a sustainable development. In: Leal Filho, W., Skanavis, C., do Paço, A., Rogers, J., Kuznetsova, O., Castro, P. (eds.) Handbook of Theory and Practice of Sustainable Development in Higher Education. WSS, pp. 201–214. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-47889-0_15

    Chapter  Google Scholar 

  36. de Oliveira, T.M., Amaral, L., Pacheco, R.C.D.S.: Multi/inter/transdisciplinary assessment: A systemic framework proposal to evaluate graduate courses and research teams. Res. Eval. 28, 23–36 (2019)

    Article  Google Scholar 

  37. Zscheischler, J., Rogga, S., Lange, A.: The success of transdisciplinary research for sustainable land use: individual perceptions and assessments. Sustain. Sci. 13(4), 1061–1074 (2018). https://doi.org/10.1007/s11625-018-0556-3

    Article  Google Scholar 

  38. Hoffmann, S., Pohl, C., Hering, J.G.: Methods and procedures of transdisciplinary knowledge integration: empirical insights from four thematic synthesis processes. Ecol. Soc. 22(1), 17 (2017). https://doi.org/10.5751/ES-08955-220127

    Article  Google Scholar 

  39. Hoffmann, S., Pohl, C., Hering, J.G.: Exploring transdisciplinary integration within a large research program: empirical lessons from four thematic synthesis processes. Res. Policy 46, 678–692 (2017). https://doi.org/10.1016/j.respol.2017.01.004

    Article  Google Scholar 

  40. Czúcz, B., et al.: How to design a transdisciplinary regional ecosystem service assessment: a case study from Romania. Eastern Eur. OE. 3, e26363 (2018). https://doi.org/10.3897/oneeco.3.e26363

    Article  Google Scholar 

  41. Wiek, A., Talwar, S., O’Shea, M., Robinson, J.: Toward a methodological scheme for capturing societal effects of participatory sustainability research. Res. Eval. 23, 117–132 (2014). https://doi.org/10.1093/reseval/rvt031

    Article  Google Scholar 

  42. Jahn, T., Keil, F.: An actor-specific guideline for quality assurance in transdisciplinary research. Futures 65, 195–208 (2015). https://doi.org/10.1016/j.futures.2014.10.015

    Article  Google Scholar 

  43. Pyshkin, E.: Designing human-centric applications: transdisciplinary connections with examples. In: 2017 3rd IEEE International Conference on Cybernetics (CYBCONF), pp. 1–6. IEEE, Exeter, United Kingdom (2017). https://doi.org/10.1109/CYBConf.2017.7985774

  44. Eanes, F.R., Silbernagel, J.M., Hart, D.A., Robinson, P., Axler, M.: Participatory mobile- and web-based tools for eliciting landscape knowledge and perspectives: introducing and evaluating the Wisconsin geotools project. J. Coast. Conserv. 22(2), 399–416 (2018). https://doi.org/10.1007/s11852-017-0589-2

    Article  Google Scholar 

  45. Belcher, B.M., Rasmussen, K.E., Kemshaw, M.R., Zornes, D.A.: Defining and assessing research quality in a transdisciplinary context. Res. Eval. 25, 1–17 (2016). https://doi.org/10.1093/reseval/rvv025

    Article  Google Scholar 

  46. Pinto, L.G., Ochôa, P.: Information science’s contributions towards emerging open evaluation practices. Perform. Meas. Metrics 20, 2–16 (2018)

    Article  Google Scholar 

  47. Hohl, S.D., Knerr, S., Thompson, B.: A framework for coordination center responsibilities and performance in a multi-site, transdisciplinary public health research initiative. Res. Eval. 28, 279–289 (2019). https://doi.org/10.1093/reseval/rvz012

    Article  Google Scholar 

  48. Kliskey, A., et al.: Thinking big and thinking small: a conceptual framework for best practices in community and stakeholder engagement in food, energy, and water systems. Sustainability 13, 2160 (2021)

    Article  Google Scholar 

  49. Gidlund, K.L.: Designing for all and no one-practitioners understandings of citizen driven development of public e-services. In: Proceedings of the 12th Participatory Design Conference: Research Papers, vol. 1, pp. 11–19 (2012)

    Google Scholar 

  50. Holzer, J.M., Carmon, N., Orenstein, D.E.: A methodology for evaluating transdisciplinary research on coupled socio-ecological systems. Ecol. Ind. 85, 808–819 (2018). https://doi.org/10.1016/j.ecolind.2017.10.074

    Article  Google Scholar 

  51. Ormiston, J.: Blending practice worlds: Impact assessment as a transdisciplinary practice. Bus. Ethics: A Eur. Rev. 28, 423–440 (2019). https://doi.org/10.1111/beer.12230

    Article  Google Scholar 

  52. Schulte, R., Heilmann, A.: Presentation and discussion of an evaluation model for transdisciplinary research projects. EJSD 8(3), 1 (2019). https://doi.org/10.14207/ejsd.2019.v8n3p1

    Article  Google Scholar 

  53. Klein, J.T.: Afterword: the emergent literature on interdisciplinary and transdisciplinary research evaluation. Res. Eval. 15, 75–80 (2006)

    Article  Google Scholar 

  54. Belcher, B.M., Ramirez, L.F., Davel, R., Claus, R.: Retraction: a response to Hansson and Polk (2018) Assessing the impact of transdisciplinary research: the usefulness of relevance, credibility, and legitimacy for understanding the link between process and impact. Oxford University Press (2019)

    Google Scholar 

  55. Stokols, D., et al.: Evaluating transdisciplinary science. Nicotine Tob. Res. 5, S21–S39 (2003)

    Article  Google Scholar 

  56. Sakao, T.: Research series review for transdisciplinarity assessment—validation with sustainable consumption and production research. Sustainability 11, 5250 (2019)

    Article  Google Scholar 

  57. Woltersdorf, L., Lang, P., Döll, P.: How to set up a transdisciplinary research project in Central Asia: description and evaluation. Sustain. Sci. 14(3), 697–711 (2018). https://doi.org/10.1007/s11625-018-0625-7

    Article  Google Scholar 

  58. Verwoerd, L., Klaassen, P., van Veen, S.C., De Wildt-Liesveld, R., Regeer, B.J.: Combining the roles of evaluator and facilitator: assessing societal impacts of transdisciplinary research while building capacities to improve its quality. Environ. Sci. Policy 103, 32–40 (2020). https://doi.org/10.1016/j.envsci.2019.10.011

    Article  Google Scholar 

  59. Hitziger, M., et al.: EVOLvINC: evaluating knowledge integration capacity in multistakeholder governance. E&S 24, 36 (2019). https://doi.org/10.5751/ES-10935-240236

    Article  Google Scholar 

  60. Danneels, L., Viaene, S.: Identifying digital transformation paradoxes. Bus. Inf. Syst. Eng. 1–18 (2022).https://doi.org/10.1007/s12599-021-00735-7

  61. Williams, S., Robinson, J.: Measuring sustainability: an evaluation framework for sustainability transition experiments. Environ. Sci. Policy 103, 58–66 (2020). https://doi.org/10.1016/j.envsci.2019.10.012

    Article  Google Scholar 

Download references

Acknowledgement

The mGov4EU project has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement No 959072.

Author information

Authors and Affiliations

  1. University for Continuing Education Krems, 3500, Krems, Austria

    Gregor Eibl & Lucy Temple

  2. Fraunhofer Institute for Industrial Engineering, Nobelstraße 12, 70569, Stuttgart, Germany

    Rachelle Sellung

  3. Johan Skytte Institute for Political Studies, University of Tartu, Lossi 36, 51003, Tartu, Estonia

    Stefan Dedovic, Art Alishani & Carsten Schmidt

Authors
  1. Gregor Eibl
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Lucy Temple
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Rachelle Sellung
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Stefan Dedovic
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Art Alishani
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Carsten Schmidt
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Gregor Eibl .

Editor information

Editors and Affiliations

  1. Delft University of Technology, Delft, The Netherlands

    Marijn Janssen

  2. Corvinus University of Budapest, Budapest, Hungary

    Csaba Csáki

  3. Linköping University, Linköping, Sweden

    Ida Lindgren

  4. University of the Aegean, Samos, Greece

    Euripidis Loukis

  5. Linköping University, Linköping, Sweden

    Ulf Melin

  6. Danube University Krems, Krems an der Donau, Austria

    Gabriela Viale Pereira

  7. University of Granada, Granada, Spain

    Manuel Pedro Rodríguez Bolívar

  8. University of Macedonia, Thessaloniki, Greece

    Efthimios Tambouris

Rights and permissions

Reprints and permissions

Copyright information

© 2022 IFIP International Federation for Information Processing

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Eibl, G., Temple, L., Sellung, R., Dedovic, S., Alishani, A., Schmidt, C. (2022). Towards a Transdisciplinary Evaluation Framework for Mobile Cross-Border Government Services. In: Janssen, M., et al. Electronic Government. EGOV 2022. Lecture Notes in Computer Science, vol 13391. Springer, Cham. https://doi.org/10.1007/978-3-031-15086-9_35

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-15086-9_35

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-15085-2

  • Online ISBN: 978-3-031-15086-9

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Societies and partnerships

Search

Navigation