Embracing diversity through mixed methods research

Mixed methods research – ‘the third methodological movement’ (Teddlie & Tashakkori, 2009) – appears to be one of those phenomena that attracts considerable interest but is rarely brought into practice, at least judging by the publications in major information systems (IS) outlets, where mixed methods studies represent only 3% of the published articles (Venkatesh et al, 2013). It is refreshing, however, to now see some increasing activity and acceptance of the approach vindicated by recent submissions to journals and conferences, in addition to recent publications (e.g. Venkatesh et al, 2013; Zachariadis et al, forthcoming). Mixed methods research is characterized by a combination of quantitative and qualitative methods within a single study. Johnson & Onwuegbuzie (2004, p. 17) define mixed methods research as ‘the class of research where the researcher mixes or combines quantitative and qualitative research techniques, methods, approaches, concepts or language into a single study’. The use of, and emphasis on, each of the two components may vary, although one often dominates. In the literature, a distinction is sometimes made between mixed methods research, which combines qualitative and quantitative methods, and multimethod research, which combines methods that may or may not share the same world-view (Venkatesh et al, 2013). A mixed method is therefore always a multimethod, but a multimethod is not necessarily a mixed method. The real strength of mixed methods is the possibility of developing meta-inferences based on a combination of qualitative and quantitative data and analysis (Venkatesh et al, 2013) – that is, developing an understanding of a phenomenon for which either approach in isolation would be insufficient. For example, a study of ‘opensourcing’ as a global sourcing strategy (Ågerfalk & Fitzgerald, 2008) developed a framework through a grounded analysis (based on qualitative interviews) followed by factor analysis (based on a quantitative survey). The framework comprised grounded categories corresponding to principal components. Parts of this framework could not be explained by the quantitative data alone but, by revisiting the qualitative analysis, a more complete understanding could be achieved.

In the following, I outline an argument for mixed methods being potentially very useful in IS research, and indicate why you should consider sending your best mixed methods research to the European Journal of Information Systems (EJIS). In doing so, I will touch upon some of the philosophical and practical issues related to mixed methods and to our discipline. However, I do not go into detail about designing or conducting mixed methods studies; an up-to-date overview of and guidelines for conducting mixed methods research in IS are available elsewhere (Venkatesh et al, 2013).

Motivating mixed methods

With close to 800 citations in Google Scholar as of 25 February 2013, Mingers (2001) has contributed significantly to the use of mixed method research and multimethod research in IS. Drawing on a three worlds ontology (Habermas, 1984), Mingers argues that, to achieve complete understanding of a social phenomenon, we need to apply multiple paradigms (or at least perspectives) to capture adequately (a) the objective (the material) world, (b) the subjective (my personal) world, and (c) the social (our inter-subjective) world. As Mingers (2001, p. 245) points out, ‘each domain has different modes of existence and different epistemological possibilities’. Understanding the objective world is oriented towards objectivism and observation, understanding the subjective world is oriented towards subjectivism and experience, and understanding the social world is oriented towards participation and shared understanding.

Habermas (1998) argues that three different attitudes – which he refers to as objectivating, expressive, and norm-conformative – map to each of the three worlds in such a way that (a) an objectivating attitude is primarily associated with the objective world, (b) an expressive attitude is primarily associated with the subjective world, and (c) a norm-conformative attitude is primarily associated with the social world. He also suggests (Habermas, 1998, p. 421) that we may ‘vary these attitudes in relation to one and the same world’ and thereby analyse the different worlds with different attitudes. Such analysis gives rise to three different rationalities, namely (a) cognitive–instrumental rationality (when the objective and social worlds are approached with an objectivating attitude), (b) moral–practical rationality (when the social and subjective worlds are approached with a norm-conformative attitude), and (c) aesthetic–practical rationality (when the subjective and objective worlds are approached with an expressive attitude). Essentially, positivist and quantitative methods are primarily associated with cognitive–instrumental rationality while interpretive and qualitative methods are primarily associated with moral–practical rationality and aesthetic–practical rationality. In a discipline such as IS, where technically implemented social systems are the main object of study (Goldkuhl & Lyytinen, 1982), such a multi-perspective outlook would seem essential (cf. Ågerfalk & Eriksson, 2006; Eriksson & Ågerfalk, 2010).

Mixed methods and paradigms may provide the tools required to tend adequately to the three worlds with their associated attitudes and rationalities, in such a way that, for example, the constituents of the subjective and social worlds (subjects, subjective experiences, subjective and collective intentions, norms, rights, social rules, and relationships) are not confused with the constituents of the objective world (physical entities and their properties) and are made accessible only by an objectivating observing attitude. Consequently, the subjective and social worlds may be studied without being forced into the basic conceptual framework of natural science and technology, which would deprive them of their subjective, social, institutional, and pragmatic dimensions. Quantitative techniques and (objective) observations could then, for example, be used in tandem with qualitative techniques and (interpretive/subjective) participative interaction to achieve a more complete understanding of the different rationalities that underpin the socio-material and socio-technical realities that affect our various IS practices. As an example, we can imagine a study that employs a randomized controlled experiment to study the effectiveness of a particular IT security protocol in an organization (i.e. a quantitative method studying the objective and social worlds using an objectivating attitude concerned with cognitive–instrumental rationality) in combination with a qualitative case study of users’ interaction with the system in their actual work practice (i.e. a qualitative method studying the social world with a norm-conformative attitude concerned with moral–practical rationality). In isolation, neither of these approaches could provide the same insight as can their combination. The key here, though, is to develop a strategy for facilitating high-quality meta-inference (Venkatesh et al, 2013), for example by letting factors found to affect the effectiveness of the security protocol be used as seed categories (Miles & Huberman, 1994) to drive the qualitative analysis of the case data. This also means that the understanding of the instrumental effectiveness of the protocol can be compared and matched with its suitability from a moral–practical perspective (cf. Ågerfalk & Eriksson, 2006) to ascertain that it both does the right things (moral–practical rationality) and does the things right (cognitive–instrumental rationality).

On a more general note, mixed methods research has been touted as important for a number of reasons, including (Johnson & Onwuegbuzie, 2004; Venkatesh et al, 2013):

• triangulation – using different methods and designs in studying a phenomenon to identify convergence and corroboration;

• complementarity – using the results from one method to clarify or illustrate the results from another;

• initiation – discovering paradoxes and contradictions that lead to reframing of the research question;

• development – using findings from one method to inform a research design involving another method;

• expansion – using different methods for different inquiry components to expand the depth and breadth of the research; and

• diversity – using different methods to identify diverging views of the same phenomenon.

This list is by no means exhaustive, and it is often the case that one particular mixed methods study design may be motivated by several of these (and other) reasons. For example, a study of enterprise resource-planning implementation (El Amrani et al, 2006) used qualitative case studies to inform the construction of a questionnaire used in a quantitative survey (development). Re-analysis of the case and additional qualitative interviews were then conducted to understand better the implications of the quantitative results (complementarity).

Paradigmatic considerations

When mixing quantitative (typically positivist) and qualitative (often interpretive) approaches, paradigm incommensurability may become a concern. The so-called ‘incompatibility thesis’ (Howe, 1988) even suggests that quantitative and qualitative research paradigms should not be mixed for this very reason. One way to address the incompatibility thesis is to subscribe to a paradigm that is inclusive enough to accommodate the inherent tension. In this regard, critical realism is often quoted as a suitable paradigm for mixed methods research (Mingers, 2001; Venkatesh et al, 2013; Zachariadis et al, forthcoming). A similar suggestion (Johnson & Onwuegbuzie, 2004, p. 17) is that pragmatism offers an ‘immediate and useful middle position philosophically and methodologically’ that, with its emphasis on finding a practically useful method to researching a specific question, may serve as an appropriate paradigmatic grounding (Onwuegbuzie & Leech, 2005; Venkatesh et al, 2013). It could thus be seen as a ‘third independent paradigmatic position’ (Ågerfalk et al, 2006, p. 5) as suggested by Goles & Hirschheim (2000). Alternatively, one may adopt a pragmatic position in a pragmatic rather than philosophical way (Ågerfalk, 2010), thereby following the suggestion (Mingers, 2001, p. 243) that it is possible to detach a research method from a particular paradigm and use it in other contexts, provided it is done ‘critically and knowledgeably’ [emphasis in the original]. With such an approach, it is more important to acknowledge the strengths and weaknesses of a particular mix of methods within the confines of a specific study than to pay attention to abstract paradigmatic logics that may not even be particularly critical in the current context.

Certainly, one can maintain, for example, a strictly positivist or a strictly interpretive world-view while conducting mixed methods research. I would argue, however, that embracing diversity by entertaining the tension between different world-views and paradigms is one of the most exciting and potentially useful aspects of mixed methods research. The design of a study's research method should always follow from the research questions. Letting paradigmatic prejudices reduce the degrees of freedom in that process may well be counterproductive and lead to less interesting and less useful results. It has even been argued that ‘mono-method research [i.e. strictly quantitative or qualitative research] is the biggest threat to the advancement of the social sciences’ (Onwuegbuzie & Leech, 2005, p. 384) because it leads to polarization and unnecessarily limited studies.

Doing mixed methods research

As indicated above, mixed methods can be used in a variety of different contexts and can play a role in very different research designs. Mixed methods research design can be categorized in terms of (a) the time order between qualitative and quantitative aspects and (b) the possible dominance of one approach (Johnson & Onwuegbuzie, 2004). That is, we can think of studies that start with a qualitative approach followed by a quantitative approach, or the other way around, and studies where both approaches are used in parallel throughout. In such studies, qualitative and quantitative approaches may have equal status, or one approach may dominate. Common scenarios include, for example, starting with a qualitative case study to develop a grounded research model that is subsequently explored by means of a quantitative survey or alternatively, starting with a quantitative study and using qualitative strategies to situate and explore the initial findings in more depth. A related issue is the use of theory in mixed methods research. Given the nature of mixed methods, a study could be inductive, deductive, or a combination. Consequently, the role of theory is contingent on the overall research design and has to be considered in relation to the research questions at hand.

In summary, at least four central questions inform the design of a mixed methods study (Creswell, 2003):

• the sequence in which the qualitative and quantitative data collection will be implemented;

• the relative priority to be given to qualitative and quantitative data collection and analysis;

• the stage of the project at which the qualitative and quantitative data will be integrated; and

• the extent to which an overall theoretical perspective will be used to guide the study.

In addition to the more traditional uses of mixed methods outlined above, a research approach where mixed methods could potentially play an important role is design science research, which has emerged as an important research orientation in Europe and throughout the world (Winter, 2008; Baskerville et al, 2011). Design science research has been described (Hevner et al, 2004; Hevner, 2007) in terms of build–evaluate cycles. Arguably, evaluation is an area that could benefit from the above-identified strengths of a mixed methods approach. As argued above and elsewhere (Gregor & Jones, 2007), if understanding an IT artefact in context requires one to understand the constituents of the subjective and social worlds and how these relate to the constituents of the objective world, then a mixed methods approach to evaluation seems appropriate. Such an evaluation could involve, for example, an objectivating attitude (to evaluate the internal technical quality by means of cognitive–instrumental rationality through automated tests) combined with a norm-conformative-oriented attitude (to understand the user experience of the artefact in its social context by means of moral–practical rationality through in-use evaluation methods).

Another area where mixed methods may play an important role is in the analysis of the ‘big data’ resulting from the massive contemporary use of social media. Whereas quantitative and highly automated analyses may report on frequencies of interaction, structures of social networks, etc., what these data and social interactions actually mean requires qualitative understanding. Furthermore (Latour et al, 2012), the new availability of digital data sets makes it possible to navigate these sets via individuals’ own connections, without having to make the traditional distinction between the level of individual components and that of the aggregated structure, thereby potentially challenging the established ground rules of social theory. Clearly, this offers an opportunity for IS research in terms of both theorizing the development and designing future socio-material practices. More generally, there should be ample opportunity for both critical and design-oriented studies of, for example, data collection through social media, profile mining, online privacy, integrity and security, and the corporate use of social media, where big data and qualitative insight could go hand-in-hand to enrich our understanding of the phenomena, in addition to deriving design guidelines and creating novel artefacts.

I am not suggesting that these are the only ways to bring mixed methods to the table. On the contrary, my ambition here has been to encourage new, innovative, and productive ways of mixing methods and paradigms to solve important IS problems and to bring new insights to our field. Meta-analyses that critically address the issues at stake and explore the pros and cons of mixed methods in relation to epistemological and ontological assumptions, as they apply to IS specifically, could also be very useful (the pragmatics of pragmatism vs critical realism would indeed be a good candidate).

Mixed methods and EJIS

In a recent editorial, the EJIS Editor-in-Chief (Rowe, 2012) shared his vision of how our journal could better serve the IS community by more explicitly acknowledging the breadth of genres prevalent in IS research. For a journal that is known to represent a distinctive European perspective on the theory and practice of IS, embracing diversity is commendable. In keeping with this vision, EJIS recently announced an upcoming Special Issue on alternative genres (Avital et al, 2012).

The upcoming Special Issue and the Editor-in-Chief both call for increased diversity in terms of genres represented in IS research. In this editorial, I have pushed for a related but different kind of diversity, namely diversity in empirical research methods and paradigms. Both kinds of diversity are important and resonate well with the objectives of EJIS. Embracing diversity through mixed methods could also serve the purpose of fostering a research tradition that emphasizes ‘critical and knowledgeable’ selection and combinations of research methods that best match the research question and empirical setting at hand, without paradigmatic blinders. With its explicit ambition to promote diversity, EJIS provides a natural forum for such research, so please consider sending us your best work and help establish our journal as the primary outlet for mixed methods in IS.

In this issue of EJIS

This issue of EJIS includes six articles. The first article ‘Digital business reporting standards: mapping the battle in France’, co-authored by Véronique Guilloux from UPEC France, Joanne Locke from Birmingham Business School and Alan Lowe from Aston Business School, traces the existent relationship dynamics and pressures between two competing business reporting and ICT data standards in France; namely the Electronic Data Interchange for Administration, Commerce and Transport (EDIFACT) and eXtensible Business Reporting Language (XBRL). The study mobilizes Actor Network Theory (ANT) to examine the impact of certain actors and events on adoption decisions made by French government bodies and institutions and the path these standards have followed. The study, among other things, shows how regulator actors play an important role as adopters in the path that this relationship follows. A new standard may then challenge the position of an existing incumbent standard under certain conditions. The study also shows the influence that certain networks and structures may have on such a standards adoption relationship, while questioning the differentiating attitudes and whether speed of development of the standard is more important than its legitimacy in relation to adopters’ decisions.

The second article ‘Top management support in multiple-project environments: an in-practice view’ by Amany Elbanna from Royal Holloway University of London, also uses the ANT framework but this time to challenge assumptions related to the top management's steady and consistent support as being a critical factor for the success of an IS project. A novel aspect of this research is that the question is raised in a multi-project setting. The study shows that top management support is not as constant as previously assumed, nor unidirectional or passively available, but rather, it is constructed through projects’ efforts to attract top management's attention. The project's actors most often change over time. Similarly, a project's continuation and success depends on its active mobilization of local networks hence the call addressed to project managers and practitioners to build and strengthen their project's local network and continue efforts, despite the lack of top management attention that may be witnessed.

Our third article, titled ‘Information privacy and correlates: an empirical attempt to bridge and distinguish privacy-related concepts’, is co-authored by Tamara Dinev and Paul Hartfrom from Florida Atlantic University, Heng Xu from Pennsylvania State University, and Jeff H Smith from Miami University, and addresses the multidimensionality of the concept of information privacy. The study mobilizes perceived privacy as a dependent variable for information privacy. Perceived privacy in the proposed research model is the outcome of two variables namely perceived information control and perceived risk. Perceived information control uses three tactics to achieve control: anonymity, secrecy, and confidentiality. Perceived risk is the outcome of the perceived benefits of information disclosure, the information's sensitivity, the importance of information transparency, and matching regulatory expectations. The model is supported and shows strong relevance using data collected from 192 responses to an administered survey.

The fourth article titled ‘Sympathy or strategy: social capital drivers for collaborative contributions to the IS community’ presented by Matthias Trier from Copenhagen Business School and Judith Molka-Danielsen from Molde University College investigates researchers’ structural patterns of academic collaboration and co-authorships using a social network perspective. It does so by taking into considering different styles and profiles of research in the IS field that also involve citation and publication preferences. Utilizing analytical dimensions suggested by social capital theory, the study shows that inter-organizational relationships form, to a large extent, a central backbone in scientific productions, whereas at the periphery, national relationships dominate. It also finds that structural and relational social capital dimensions were perceived as being critical. Finally, the study also establishes that a low level of network centrality is closely related with a topic-oriented disposition.

The fifth article ‘A method for taxonomy development and its application in information systems’ written by Robert C Nickerson from San Francisco State University, Upkar Varshney from Georgia State University and Jan Muntermann from University of Göttingen advances a methodology for developing a taxonomy adequate for the IS field. The approach for taxonomy development follows a design science approach and starts by setting meta characteristics, then following an iterative process of empirical-to-conceptual and conceptual-to-empirical pattern new dimensions under a particular taxonomy are created. The taxonomy then consists of set of dimensions, each consisting of mutually exclusive and collectively exhaustive characteristics such that each object under consideration has one and only one characteristic for each dimension. The article also suggests objective and subjective ending conditions that would ensure that the new dimensions introduced under a particular taxonomy meet the criteria and conditions desired. The taxonomy development method is illustrated using the case of mobile applications.

The sixth and final article in this issue, ‘Can we have fun @ work? The role of intrinsic motivation for utilitarian systems’, is co-authored by Jennifer E Gerow from Virginia Military Institute, Ramakrishna Ayyagari from University of Massachusetts, Jason Bennett Thatcher, and Philip L Roth from Clemson University. It runs a meta-analysis over 185 user acceptance studies to question whether the nature of the system influences intrinsic motivation's relationship with users’ perceptions, intentions, and use of that system. According to the authors, the system could have either a hedonic, utilitarian, or mixed nature. The impact of both intrinsic and extrinsic motivation over perceived enjoyment influences the perceived ease of use. Hence, the recommendation to system developers is to pay attention to such motivational features in the design phase that engage users across all system types. This is particularly relevant since the meta-analysis revealed that the relationship between intrinsic motivation and the traditional TAM constructs was similar across system types.