Keywords

1 Introduction

Healthcare systems worldwide are increasingly under pressure, explained by our aging population, increasing numbers of people living with long-term chronic conditions, and spiraling costs of healthcare provision [13]. In the face of these challenges, there is widespread agreement on the urgent need to develop and improve the efficiency in healthcare. eHealth services, i.e. health services provided to people via Internet, promise to help solve the demand by improving the quality and effectiveness of care. However, these services are still to be considered new, and therefore they suffer from a range of technological, human, and organizational issues which pose challenges for the provision of quality healthcare. In practice, there is little time to reflect and draw conclusions around these complex issues and to increase our understanding of the context in order to improve the design of the eHealth services. In other fields, critical incidents (CIs) have been used to analyze failures of procedures or human errors in order to reduce risks in the future. Based on this, we appropriated the CI technique for IT development in health and looked at critical incidents as a basis for future directions in the development of eHealth services. This paper consequently describes the full-day workshop format, in which real-world CIs, representing development, healthcare organization, and patient perspectives, can be used as a tool to critically reflect on and analyze design cases of eHealth services [3]. The main focus hereby is on the method itself, its applicability for use as a workshop format, and the value of using CIs as a tool to inform the design of eHealth services.

2 Background

The two key concepts relevant for this paper are eHealth services and the Critical Incident Technique.

eHealth Services: eHealth refers to health services and information delivered or improved through the Internet and related communication technologies in order to enhance healthcare locally, regionally, and worldwide [4]. By breaking down the barriers of time and place, eHealth brings people and resources together to deliver healthcare services more efficiently [9]. Thus, eHealth enables changes in healthcare practices which can have an impact on patients’ lives. For instance, the ability to access their electronic health records (EHR) can help patients to understand their medical issues, prepare for their next visit to their doctor, and help them to feel more in control of their care [19].

eHealth poses challenges for designers, in part because it changes methods of diagnosis, monitoring, and treatment [10] and creates interactions between actors, organizations, and systems that did not exist in conventional healthcare settings [12]. It may also prove more difficult to identify errors that put patients at risk, e.g. with electronic prescriptions [1]. These services are expected to be used by patients and professionals who need access to health information, from public health practitioners to administrators. But including a large number of heterogeneous end-users, especially for patients and other non-medical user groups, makes the development of eHealth services a complex task [2]. For instance, individual requirements can be in conflict, e.g. requirements with regard to privacy and data protection [24] as well as regarding how and when data is accessed and by whom [8]. Professional requirements can also cause conflicts, e.g. policy makers’ versus clinical users’ needs to manage the same data [1].

Many of the promises regarding how technological achievements will transform healthcare have to date fallen short of expectations [11]. As eHealth services are still to be considered new, the degree of uncertainty regarding end-user needs is high [20]. Faults or errors are unacceptable and systems must be reliable, dependable, and interoperable [22]. In some cases eHealth implementations have had unanticipated negative consequences that put patient lives at risk [1]. Thus, when developing eHealth services, the involvement of real users, i.e. patients and other stakeholders, is desirable [18]. With patients, who are possibly ill or weak, this can be particularly challenging [7], and in some cases may not be possible. However, involving proxies to represent the user is also problematic [15, 23]. Thus, to improve healthcare practice where eHealth is used, new methods are needed to involve the experience of patients and relatives as well as to learn from previous projects. While studies abound reporting on failures or poor usability of health information technology, to date there is a lack of research investigating failures related to eHealth services (i.e. health services delivered over Internet). Specifically, the contextual causes of these failures as viewed by the non-medical user groups have rarely been studied.

The Critical Incident (CI) Technique: In the area of healthcare, the Critical Incident (CI) Technique has proved valuable in analyzing failures (see e.g. [25]). The CI technique was originally used to analyze failures of procedures or human error in fields like aviation in order to reduce risks in the same environment in the future through changes to system design [5]. Its use subsequently spread to health, education, and social work where it was applied with a shift of focus from failure examination to critical reflection [14]. The importance of critical reflection in practice is discussed by Donald Schön as a way for practitioners to make their tacit knowledge visible so that it is available for deliberation [21, p. 61]. Similar to Schön, Fook & Gardner discuss current challenges to professional knowledge, which they identify as the key reason for the need for critical reflection [6]. Some of the points they emphasize are that a) contexts are so changeable that practitioners need to continuously reassess their knowledge in relation to the context, and b) current contexts of practice are characterized by risk, uncertainty, changeability, and complexity [6, p. 66]. The critical reflection model developed by Fook & Gardner [6] makes use of the CI technique in that groups reflect on specific examples of their practice experience. Each participant presents and reflects on their CI, which they refer to as “something (an event) that happened to a person that they regard as important or significant in some way” [6, p. 77]. In a two-stage process group members use questions to help elicit embedded assumptions (Stage 1) and, through further questioning and dialogue, help each other “derive changed practices and theories about practice that result from their reflections” (Stage 2) [6, p. 73]. In the workshop described in this paper, eHealth designs were evaluated by appropriation of the critical reflection model to learn from patients’ real stories and case studies through retrospective meta-analyses, and to inform design through joint reflection of understandings about the users’ needs and issues for designers.

3 Workshop Design

In the following, we describe the design of a workshop in which researchers, practitioners, and patients were invited to contribute with a CI in relation to eHealth services for patients in advance, as it was used in a conference [3]. Inspired by the critical reflection model [6], the workshop organizers requested not only to describe the CI, but also to answer the following questions:

  • What assumptions were made about the stakeholders, problem, or situation?

  • What were the consequences of this incident?

  • What could be done if this or a similar incident would occur again in the future?

The two stages of the original critical reflection model [6] were included in the workshop design. Inviting the participants to reflect on their CI by answering the specific questions in their position paper was Stage 1, whereas the dialogue in the group during the workshop constitutes Stage 2.

To promote active participation during the workshop, the presentation of the CIs was kept quite short and was done in an unusual manner: authors of the accepted position papers were asked to prepare a poster that contained background, problem environment, outline of the CI, and the most important aspects of why this incident was critical, that were then presented one at a time. This was in part due to the conference, which encouraged workshop organizers to make full use of the workshop format by prioritizing debate and joint action, instead of having for example “mini-conference”-style paper presentations [17].

During each presentation, the rest of the group made use of colored sticky notes to write comments related to the CI. For this activity four broad categories were introduced: Enablers (green), Barriers (orange), Learning Opportunities (yellow), Other (pink). These categories were purposively left quite open in order to prevent constraining the participants, while at the same time supporting them to think in more than one direction. After each presentation, the group members could ask further questions, engage in discussions, and finally attach their notes to the respective poster (see example in Fig. 1, left). Thus, the posters were used not only as a presentation medium, but also as an artifact for analysis and discussion throughout the workshop.

Stage 2 began after the presentations and here the participants were asked to make use of comments on the sticky notes and to discuss possible implications for design, which can be seen as a way to “derive changed practices and theories about practice that result from their reflections” [6, p. 73], as suggested by the original critical reflection model. The discussion was carried out in smaller groups, in which the sticky notes were discussed and moved from the posters to large sheets (size A1). This helped to connect them visually to the respective design implications found. Towards the end of the workshop the two groups presented and explained their findings. The workshop concluded with a short feedback session where the participants were encouraged to discuss their views of the method used for the workshop.

4 Experiences from the Workshop Conducted

In total, nine participants attended the workshop, of which eight were affiliated with a CI. As explained in the method section, preparing the CI and tentatively answering the questions constituted Stage 1. The workshop included five CIs. Two described incidents from the development perspective: a digital collage in a care home, which was perceived quite different by users than was intended by the designers; a sensor-based telecare monitoring system, of which many services were not used although the design was based on recommendations of the target group. Three CIs were related to the patient experience: patients and relatives reading the EHR; patients who lack access to their EHR; patients not being involved in discussions about their care during multidisciplinary team meetings. All presentations were followed by lively discussion, illustrating that the format worked well. After each presentation all of the participants contributed sticky notes with thoughts and ideas (Fig. 1, left). By the end of the presentations all posters had comments attached that covered all four areas (enablers, barriers, learning opportunities, other). A few comments in each category were repeated on multiple posters, but most comments were unique. For the discussion in Stage 2, the participants were split into two groups. The choice of the groups was based on the two focus areas of the different CIs submitted (i.e. patient experience and development perspective). During this stage numerous implications for design were identified. Figure 1 (right) exemplifies the result of one group discussion.

Fig. 1.
figure 1

Use of sticky notes: After presentation (left), after group discussion (right).

Full size image

At the end of the full-day workshop, the participants were invited to reflect on the workshop and on the use of CIs as a tool to inform eHealth design. The provided feedback is summarized in the following:

Workshop design and realization: As explained in the method section, the workshop was organized to support active participation. This was perceived as positive and stimulating, especially because listening to presentations was expected to fill the following conference days and a workshop should therefore not consist solely of presentations. Using posters as a presentation medium, which were then used for analysis and active discussions, was considered constructive. Time-keeping for the active discussions during the workshop proved to be very difficult. This would have been even more difficult if more CIs had been included, as the organizers had intended. The participants perceived the workshop to be fairly relaxed and did not mind that the schedule was not strictly followed.

Critical incidents as a tool: The conclusion that evolved through discussion was that although the focus of the CIs might have been different, there are some problems and experiences that these projects have in common. For example the importance of including a broad diversity of users and stakeholders, or the different views on data taken by healthcare professionals and patients. For both of these, one implication identified was the need for checking with users, iteratively, starting early in the project. While it is seldom that failures are reported at conferences, much can be learned from analyzing and discussing these incidents. In addition, the value of this format for use within design teams was discussed. The CI format makes it possible for other people to access individual projects, to understand what is happening, and to comment on them. Thus, it is possible for participants to provide valuable comments from the outside in a relatively short period of time. Although Stage 2 focused on design implications, many of the projects were completed, so rather than identifying novel design implications the primary value for the participants themselves was found in learning about the area, gaining insight, and getting a deeper understanding.

The call for contributions: Some participants reported difficulty when preparing their CI/position paper after reading the instructions in the call for contributions. The format was unusual and it seemed like a “force fit”. The participants’ feedback brought to light that the call implied a strong focus on the patient perspective, of which the organizers were unaware. This implication, and the difficulties experienced when fitting the content into the CI format, almost led some researchers to refrain from submitting their contribution. However, the CI format was also seen as an opportunity to look at a project or a case from a different perspective. One participant related that the process of re-examining and re-framing the project into the CI format led to a new perspective on it.

5 Discussion

In the following section the various views on CIs noted, the way the workshop helped the participants, and comments on the workshop format are discussed.

Workshop format: The workshop format proved to be very successful and can be recommended to others working in the area of eHealth. A couple of organizational factors were key to the success. Firstly, limiting the topic to “eHealth services for patients and relatives” ensured all participants had a common knowledge base from which to discuss, even though when preparing some participants felt it was a force fit. This would be less of a problem if the workshop was held for a specific project. Secondly, using posters with a lot of graphical elements, rather than slideshows, reduced the presentation of each project to the essential elements and also provided a place on which to place the sticky notes with the comments. The four comment categories of Enablers (green), Barriers (orange), Learning Opportunities (yellow), Other (pink) worked well. Rather than limiting discussion to positive enablers and negative barriers, adding learning opportunities supported taking a view to future projects. The other category was sometimes used to draw parallels between projects. Using different colors from the start assisted communication among participants, saved time in that they didn’t have to be categorized, and supported the structure of discussion in Stage 2.

Only after the submission deadline for the position papers, the organizers got an overview on the range of critical incidents described. Because the workshop was held at a conference, it was also difficult to attract practitioners and non-academics, whose input is invaluable. Furthermore, some of the position papers were written by more than one author, and it was unclear to the organizers until the day of the workshop, how many of the authors would attend. These aspects left lots of room for uncertainty, which had to be dealt with flexibly.

Time is often an issue in workshops. A lot of time was saved in this one by having participants prepare in advance by identifying and exploring their critical incident. If patients and caregivers are included, this activity could be done at the start of the workshop to reduce the effort required for the preparation. However, this may reduce the depth of the reflection and would deny participants the opportunity to look at others’ CIs in advance.

There are some pitfalls to the Stage 2 discussion. Splitting the group, as was done here, may make it difficult to understand the intended meaning of some comments, since not all authors were present in either group. Participants thought that starting with barriers would have generated negativity in comparison to starting with enablers, as was done here.

Supporting projects moving forward: In practice, the participants provided valuable and quite detailed comments about others’ projects about which they knew little. This was possible because the workshop had a focussed topic, and because each project focussed on a single CI. The focus made it easier for presenters to reduce some complexity, and having a concrete example made the problems easier to understand.

Participants gained valuable insights from the workshop. For example, based on a critical incident about a patient who needed access to their EHR urgently, workshop participants came to discuss in which way the structure of an EHR needs to be different for doctors and patients. The discussion made certain beliefs visible, challenged held assumptions, and helped identify new possibilities for collaboration between different stakeholders. Furthermore, it helped participants to see more clearly which uses were intended and which were appropriated. At the same time, it highlighted the limits to our understanding about potential future usage of eHealth systems, especially in early phases of the design process.

There were similarities between both the problems faced in the projects and the solutions proposed. Although the workshop started by looking at individual projects, it ended up identifying issues and design implications relevant for eHealth more generally. On reflection, the issues identified were not trivial, further demonstrating the value of the workshop. In addition, the way the issue was discussed during the workshop made it easy to apply to a specific project. This meant there was a benefit even to participants who did not present a CI and those whose projects were already completed. It also contributed to a sense of a shared vision for the future.

Different views on Critical Incidents (CI): The CI technique was originally designed to investigate problems, e.g. disorientation in pilots [5, p. 329], in order to inform design. More recently it has also been used as a method to analyze needs in User Centered Design (UCD) [16], and is also established in social work and health sciences to improve professional practice [6]. The conducted workshop demonstrates the value of CI for the development of eHealth more specifically. In this workshop people examined CI from two different perspectives. Some chose CIs from a patient’s perspective to understand the needs. Others chose the CIs from the perspective of the developers, i.e. the problems faced during the development itself. All participants, regardless of perspective they took, found they gained understanding from the process. For the Stage 2 discussion, the developer and patient perspectives were separated, so that each group looked at contributions taking the same perspective. Also retrospectively, separating these perspectives for developing solutions makes sense, as the groups focussed on different issues: the developer perspective focussed more on the development process; and the group with the patient perspective more on specific design issues. It may be advisable to specify which viewpoint participants should take in order to promote discussion - but with the trade-off that only a narrower scope can be covered.

6 Conclusions

Basing this workshop around CIs is a novel approach that proved valuable. The workshop helped participants (researchers and designers) gain valuable insight into a variety of different eHealth projects, which they can make use of also in future projects. We recommend this method to others working in eHealth design. This type of CI workshop can be used to evaluate systems for the purpose of producing suggestions for the improvement of the design itself, but also to evaluate the development processes and/or to gain understanding about the application area. Developing an understanding of the area is of particular value in eHealth, where the introduction of systems and services enable new and unanticipated interactions that are difficult to analyze thoroughly in advance. In addition to the application with people from the development team representing real incidents as described here, as a next step we propose using the same format with actual stakeholders (such as healthcare professionals, patients, relatives or other non-medical users), possibly together with members of the development team. Here, the preparation tasks should be adapted to reduce the effort for the actual stakeholders (e.g. refrain from the position paper and concentrate on the presentation of the CI). In addition, it would be even more important to address the challenges regarding participant recruitment proactively, to ensure all relevant groups are represented.