Keywords

1 Introduction

Researchers who study human work processes often use qualitative theory and methods, including grounded theory, cognitive work analysis, and ethnography, to generate rich descriptive models of human behavior in context (e.g., Vicente 1999). In contrast, experimental psychologists typically do not receive training in qualitative induction, nor are they likely to practice ethnographic methods in their work, since experimental psychology tends to emphasize generalizability and quantitative hypothesis testing over qualitative description. However, qualitative frameworks and methods from anthropology, sociology, and human factors can play an important role in enhancing the ecological validity of experimental research designs.

This paper describes elements of work domain field research conducted as part of Sandia National Laboratories’ Pattern ANalytics for High Performance Exploitation and Reasoning (PANTHER) project, an internally-funded effort to develop algorithms, software and visualization environments that will enable national security analysts to detect, characterize and communicate meaningful geospatial and temporal patterns in large, complicated remote sensing data. A key PANTHER goal is empirical identification and experimental validation of the perceptual and cognitive skills that characterize effective geospatial pattern analysis in high-throughput work environments. This information is considered a critical source of requirements for developing, implementing and evaluating new visual analytics technologies aimed at balancing human detection skill with automated analysis of threat patterns over greater geospatial and temporal domains (see discussion in Jian et al. 2000).

Given the significance of visual inspection in a wide variety of national security work domains, experimental studies examining visual search strategies, skill acquisition and factors influencing performance are quite important in realizing this larger project goal. However, such research should replicate key parameters of the work environment to optimize the ecological validity and applicability of findings. Experimental, laboratory-based studies of human visual attention tend to rely on batteries of detection tasks that use standardized stimulus sets (i.e., identifying a unique Q in a field of distractor Os), which bear little resemblance to the real-world work of visual inspectors. This lack of conformity challenges the ecological validity of experimental findings; for example, by failing to account for the importance of human memory in visual search strategy and target detection; or by underestimating the complexity of the perceptual environment when designing information displays (Shore and Klein 2000; Burke et al. 2005). In contrast, human factors and industrial engineering researchers who study visual inspection often collect data using study designs that mirror real-world work contexts; for example, observing and evaluating the performance of industrial inspectors as they examine an aircraft for anomalies associated with structural defects (Drury et al. 1997, 2004; Hong et al. 2002; Wenner et al. 2003). Yet developing realistic models of work environments to inform experimental designs that also meet the standards required high experimental validity is both methodologically and conceptually challenging.

2 Field Studies in the Analytic Workplace

Over the past decade, Sandia National Laboratories has expanded capabilities in the study of perception and cognition among professionals whose work predominantly consists of visual search and anomaly detection. As Matzen et al. (2015) point out, studies that examine issues such as variability in search performance using real world stimuli, or the acquisition of search skills among “professional searchers,” are surprisingly rare. This is despite the fact that visual anomaly detection is critical for detecting, characterizing, and taking action against a broad range of problems, from evidence of persistent threat activity on corporate information networks, to recognizing evidence of emerging neoplasms in radiological scans. The complex cognitive and perceptual activities of visual search comprise a socially, economically, and even politically critical skill domain that deserves attention, if only to ensure the design of work environments that minimize unnecessary sources of load and stress that could induce error. In addition, studying visual search in context presents new opportunities to appreciate the role of environment and experience in the acquisition, maintenance, and evolution of perceptual and cognitive skill.

Yet developing domain-faithful study designs that allow at least some generalization of research findings is not an easy task. Work domains can be difficult to identify and access, which is not surprising considering how sensitive some of the information may be. In addition, measures of human performance on a visual inspection task may themselves constitute sensitive information for the company or government agency. Thirdly, real-world visual search and analysis workflows tend to be highly idiosyncratic, even when inspection goals are nominally identical. For example, academic, industrial and government institutions in the United States employ many research and analysis teams in which people scan remote sensing imagery for evidence of landcover changes. However, across these institutions, the tools, methods, training, physical environment, and products vary in ways that make comparative studies quite challenging.

To address this gap, we have explored the use of qualitative field methods, particularly ethnographic methods from cultural anthropology, to understand how people accomplish visual search-related work tasks. During the past three years, we have been focusing on imagery analysts who work with SAR image products in a high-pressure, high-throughput national security work environment. Our work was motivated by the need to determine if new electronic image products, interaction models, and graphical representations being developed by our PANTHER counterparts could be effective in helping imagery analysts detect and characterize a greater range of signature types using larger collections of electronic imagery. Doing so required empirical characterization of the existing workflow, including individual strategies for detecting, identifying, and making decisions about the meaning of anomalous artifacts in SAR image products. To ensure that our experimental studies captured relevant elements of the real-world analytic workflow, we invested roughly 18 months of work characterizing the SAR image analysis process; a year of this work was completed before designing and implementing the experimental studies described in Matzen et al. (2015).

2.1 Ethnographic Field Methods: From the Village to the Corporation

Ethnography, literally the “writing of culture”, is the hallmark methodology of cultural anthropology. This methodology comprises a number of methods, including participant-observation, interviews, and the collection and documentation of domain-relevant artifacts that inform a holistic account of collective ways of knowing that knit a group of individuals into a socioculturally coherent whole.

At first blush, the relevance of ethnography for design of perceptual and/or cognitive experimental studies may not be obvious. However, since the late 1980s, a number of fields have embraced qualitative research, including approaches associated with ethnography, to address the knowledge gaps associated with quantitative paradigms, including the design of quantitative data collection strategies. This trend is particularly apparent in applied research domains where research findings are being used to influence program, technology, policy, or organizational design decisions. For example, healthcare evaluation researchers now commonly use a blend of quantitative and qualitative methods, both serially and in parallel, when examining how variations in organizational structure, hospital service delivery models, and staff-patient interactions influence health outcomes (Ostlund et al. 2010; see also Bastien 2008).

Arguably, however, the vanguard of mixed-approach methodological innovation is located in the consumer technology industry, where since the late 1980s qualitative inquiry has become core element of design practice. The work of anthropologist Lucy Suchman is often cited as inspiration for the emergence of user-focused design paradigms. As Suchman’s work emphasizes, technological artifacts reify tasks and goals in ways that are intended to support or facilitate human enactment of those activities (Suchman 1987). Ball and Ormerod (2000) locate Suchman’s theories of artifacts in the earlier work of psychologist Herbert Simon, who described the design of physical artifacts as the archetypical externalization of human cognitive work and problem-solving. When people are creating artifacts for themselves, the reification of tasks, processes and goals is a relatively straightforward matter: we are creating things that meet needs we understand intimately and implicitly. However, technology designers often work at a distance from the activities they intend to influence. This distance puts them at a disadvantage when it comes to creating artifacts that are “ready to hand,” to use Martin Heidegger’s phrase (Macaulay et al. 2000).

Ethnographic field methods, which emphasize close attention to the particularities of human practice in place – that is, in historically, organizationally, and geographically bounded contexts - are widely understood to offer a framework that can help designers gain insight into the implicit characteristics of the human activity they seek to influence. But what research practices constitute “ethnography,” and how they are properly exercised, has long been a matter of methodological debate within anthropology. These days, ethnographic practice is also an active topic of debate among practitioners who are extending anthropology’s theory and methods beyond the discipline’s traditional focus on the social life of the non-Western others.

Ethnography can feel like a frustratingly open-ended, perhaps unending process of iterative observation, note-taking, memo-writing, and qualitative coding, punctuated by rounds of semi-structured interviews with domain natives. Cultural anthropologists are trained to engage in long periods (typically a one-year cycle of activity) of inductive, iterative exploration and documentation within a community. Deep engagement with the lived experience and subjective accounts of field interlocutors enable the anthropologist to identify critical events, issues and topics. However, this relatively open ended commitment to data collection does not translate well to the project-and-product oriented organizational culture of Western industrial and government work environments. Applied, collaborative research typically requires the anthropologist/fieldworker to provide team members with regular data and information projects to inform other project activities, such as experimental data collection.

At Sandia, to ensure that our ethnographic techniques yield relevant and timely observations about the work environment under study, our team structures observational activities using elements from two well-documented methodological frameworks, namely Cognitive Work Analysis (CWA) and Cognitive Task Analysis (CTA). We discuss each of these below, specifying how we have incorporated elements of both CWA and CTA into the planning, implementation and documentation of observational research in high-throughput work domains. Before doing so, however, we provide a brief overview of the SAR technologies so the reader can appreciate the challenges of studying visual cognition in this domain.

3 Synthetic Aperture Radar and Imagery Analysis

Synthetic Aperture Radar, or SAR, is a type of active remote sensing that uses pulses of energy to create complex, two-dimensional electronic images of a scene. Because SAR is an active sensing system – i.e., the radar provides its own source of illumination – SAR systems complement established passive sensing systems, such as those operating in the near-infrared or optical range of the electromagnetic spectrum. SAR is excellent for generating broad-area, high-resolution images of terrain features under a wide range of weather conditions. They are also highly sensitive to changes in terrain features and can be used to generate detailed information about trends and events associated with weather, animal, or human activity (more information on SAR systems is available at Sandia National Laboratories’ public website on SAR systems, www.sandia.gov/radar/what_is_sar/index.html). SAR images are formed using sophisticated image formation algorithms that extract and represent different types of information in electronic format, usually on high-resolution optical displays, for human inspection.

Although SAR technologies are among the most sophisticated of today’s electronic imaging systems, analysis of SAR imagery still relies heavily on human perceptual and cognitive engagement to detect, recognize and characterize signatures of interest in rendered scenes. Organizations that use SAR imagery in their work typically employ teams of SAR analysts who are specially trained to read SAR image products, which have unique visual artifacts due to the way that SAR systems are configured and flown. At first glance, SAR imagery looks a lot like a black-and-white optical photograph, but closer inspection will reveal (among other things) oddities in spatial relationships and dark shadows that may seem to be cast by the sun, but are actually aligned with the position of the radar.

Understanding how SAR imagery analysts become skilled in reading and interpreting these features is important if projects such as PANTHER are to augment human visual skill with automated systems that enhance key features while minimizing sources of clutter and noise. What elements should be enhanced, however, depends on the problems that people are actually trying to solve using SAR image products. Even SAR analysts working in the same organization may approach their inspection tasks very differently, depending on the mission and context for which the imagery is being collected. Environmental monitoring, for example, may require an analyst to look for subtle changes in ground elevation using imagery collected over hundreds of square miles of terrain, on a monthly basis. In contrast, an analyst looking for evidence of illicit human or drug trafficking along a contested border in the very same region might search for activity signatures generated on a much shorter timescale, perhaps over a few tens of square miles. To complicate matters, SAR waveforms are data-rich and can be processed into a portfolio of image products that highlight and/or minimize different types of scene features, which are variably useful depending on the analyst’s goals.

In summary, SAR imagery analysts have access to wide range of image products that can be accessed in different order and/or resolution to support the detection and characterization of a wide range of mission-relevant information signatures. Factors related to the context of SAR imagery analysis work can therefore introduce a significant source of variability in the design of empirical data collection activities aimed at understanding how SAR professionals learn to navigate the unique, often confusing visual artifacts in SAR imagery.

4 Structuring Ethnography Using Cognitive Work and Task Analysis

Approximately four years ago, a research colleague at Sandia National Laboratories approached our team with a question: could we help her evaluate the usefulness of a new image product for a SAR image analysis task? At the time, our small team consisted of a cognitive neuroscientist, a physicist, and an anthropologist (the author). We had no experience with SAR image products; and although the anthropologist had recently completed a year of field research among professional imagery analysts, the project in question was on a much tighter timeline. We needed to quickly develop familiarity with the technology, the mission space, and the professionals doing the work. To do so, we turned to CWA and CTA for guidance in bootstrapping ourselves to a necessary-and-sufficient understanding of the SAR work domain, and we have been incorporating elements of these frameworks into our research activities ever since.

CWA and CTA are complementary frameworks for studying, respectively, a domain of work activity, as well as individual workers’ strategies for accomplishing key tasks within that domain. CWA has its origins in the ecological approaches to work first articulated by Jens Rasmussen and colleagues in Denmark in the late 1980s, and later elaborated by design researchers including Vicente (1999), Bisantz and Burns (2008) and Naikar (2011). What these practitioners share is an emphasis on holistic study of human problem-solving activities within the constraints of a work domain. These constraints span the material, ideational, purposive, communicative, organizational, and skill/knowledge elements that collectively constitute meaningful activity within the domain. CTA, in contrast, aims at detailing how an individual or team of individuals access and deploy knowledge, skill, and external resources and artifacts to accomplish critical elements of work within the domain under study (Clark and Estes 1996; Crandall et al. 2006). Used together, these frameworks can guide the collection of behavioral data to document individual, team, and organizational approaches to problem-solving in the context under study.

A deeper discussion of the theories, methods, and impact of both CWA and CTA is beyond the scope of this paper. Instead, we are interested in discussing how we adapted elements of these frameworks to inform the design of the visual search studies described in Matzen et al. (2015). In point of fact, application is one of the biggest challenges for both CTA and CWA, whose advocates prescribe implementation with significant rigor and detail. Holism is neither cheap nor easy, and as Naikar has pointed out, even experienced social, cognitive and behavioral scientists can be put off by CWA’s conceptual complexity, jargon, detailed representations, and required depth of inquiry. In a very real sense, CWA and CTA are burdened by the very prescriptive detail that ethnography lacks, and therein lies an opportunity: we assert that CWA and CTA frameworks may be selectively applied to bring structure, efficiency, and closure to ethnographic observation.

4.1 A Quick Explanation of Work Domain Analysis

Our field studies over the past four years have examined visual search among SAR imagery analysts working in two different domains. Both groups support similar missions, but each group uses different image display tools, auxiliary sources of data, and relies on different SAR image products in their work. Characterizing both domains was necessary to support PANTHER goals, as the analytic algorithms under development are intended to support the work done in each domain. However, we did not have much time: in total, we spent approximately 18 months doing observational work with workers employed by the SAR mission that we were studying, but we had to provide updated observations to our team counterparts on a monthly basis.

To structure our work, we relied heavily on two of the five core research activities prescribed under CWA: Work Domain Analysis and the Decision Ladder. In this paper, we focus on Work Domain Analysis, which systematically decomposes work into five interwoven layers of detail, starting with the artifacts that comprise the domain’s material resource base and hierarchically linking these artifacts through the processes, functions, values and priority measures, and functional purpose of the domain. Figure One is a conceptual sketch of the hierarchical representation that this line of inquiry creates. The base of the pyramid, Level One, consists of all the artifacts that the domain professionals need to do their work. The top of the pyramid, Level Five, succinctly states the domain’s raison d’etre; that is, why people created it and what purpose it fulfills in the world. In our experience, these levels are relatively straightforward to elaborate. The middle levels are a bit more challenging to understand, which is why we have labeled them with illustrative questions in Fig. 1, instead of the conceptual labels that CWA texts use.

Fig. 1.
figure 1figure 1

Simplified nominal work domain analysis hierarchy

Full size image

To motivate explanation of this framework, consider a SAR work domain that we will call “Landcover Change Monitoring,” or LCM, whose analysts are responsible for characterizing changes in land cover associated with agricultural activities and weather. We may begin our inquiry by elaborating Level One of the hierarchy by seeking information about the tools, technologies, data, information and other material/information resources used in the work. Good sources for this information include training materials, software documentation, interviews, and observation sessions. As we populate Level One with artifacts, we will be learning how people use these things. This information is represented in Level Two of the hierarchy, which captures the processes that rely on Level One’s artifacts. Level Two answers the question, “In what activities do people actually use artifacts?” For example, an LCM imagery analyst may have a desktop computer, a display monitor, a mouse, a keyboard, a server connection, image viewing software, and a file of SAR images on her local drive. The hypothetical process “Open this week’s SAR images from C:” requires all these artifacts, except the server connection; i.e., her local computer, monitor, mouse, keyboard software, and locally stored images are necessary and sufficient to perform this process. Thus artifacts define a process, while a process endows artifacts with value.

In a similar fashion, the processes in Level Two are executed in service of the functions comprising Level Three. To elaborate Level Three, we ask, “Why do these processes matter? What purpose do they serve?” In our hypothetical LCM work domain, we might find that the simple process of retrieving images from a local store is one of three processes that our landcover analyst performs as part of the function, “Evaluate recent changes in agricultural activity”. Note that the functions of Level Three can be described to a necessary and sufficient approximation in terms of the processes comprising those functions. Similarly, our LCM imagery analyst only performs those processes because they enable her to complete a key function of the domain; functions endow processes, and therefore artifacts, with value.

It is between Levels Three and Four that the Work Domain Analysis hierarchy conceptually links the internal activities of the domain – the regular, mundane analysis tasks described above – to the larger world in which the domain is embedded. To wit, Level Four asks the researcher to consider indicators of the state of the domain with regard to its raison d’etre. The functions of Level Three generate products, outcomes, knowledge, data, information, communications, etcetera. These products contribute to outward evidence that the domain is working (so to speak) as intended. The nature of this evidence is summarized in Level Four.

For example, our LCM analysts may tell us that, “Evaluating recent changes in agricultural activity” is one of the critical functions that they perform. They generate a number of reports and written products that the SAR domain provides its stakeholders. Regular issuance of high-quality reports could be an indicator of the LCM domain’s performance in Level Four. Issuing reports demonstrates that LCM analysts are indeed producing knowledge and documents that fulfill the domain’s purpose. This overall purpose would be represented on Level Five – in this hypothetical case, our LCM domain is responsible for producing knowledge about land cover changes associated with weather and agricultural activity in the region of interest.

4.2 From Work Domain Analysis to a Representative Task

The process described in the hypothetical example above is similar to the one that we followed when investigating the work domain of the SAR imagery analysts that we were tasked to study. Over the past four years, in the context of PANTHER and two earlier research activities that led up to the PANTHER project, our team has conducted extensive qualitative research with imagery analysts and other domain professionals. We have interacted with over fifty professionals performing various roles in the SAR imagery analysis domain under study. Our data have come from observing imagery analysts reviewing analysis products for completeness and correctness; open-ended interviews with system designers, users, and imagery analysts; and teach-aloud interviews with imagery analysts in both domains. In addition, we attended the SAR program’s introductory classes and practice activities, attended approximately one year’s worth of program team meetings, and observed imagery analysts participating in expert/novice paired training sessions in the program’s training center.

The prescriptive mapping of CWA’s Work Domain Analysis was extremely useful in helping us organize and summarize our research findings into a structured representation of the domain. Importantly, it also provided a starting point for our data collection interactions with the domain’s professionals, insofar as we began by asking them for assistance in creating a comprehensive inventory of the software tools, hardcopy/paper resources, electronic databases, and hardware they needed to do their jobs. Most people can understand the need to create an inventory, so this activity was a good ic.ebreaker. Moreover, in the process of identifying an artifact, people often describe how the artifact is used and why it is important – generating information that populates other layers in the CWA hierarchy illustrated in Figure One (Fig. 2)

Fig. 2.
figure 2figure 2

Task elements extracted from WDA representation

Full size image

The process of developing our WDA mapping also enabled us to identify key tasks and the artifacts and processes associated with those, which is very useful in developing a protocol for Cognitive Task to focus on the details of individual strategies for completing a particular element of domain work. In our experience, tasks can be derived from Level Four’s functional elaboration of the work domain. The artifacts and processes that support the function should be necessary and sufficient for a knowledgeable domain professional to perform the task. For example, in Figure Two, we have highlighted a function at Level Three, along with the processes and artifacts on Levels Two and One that are required for the function to be performed. Level Four tells us about what the task generates and why it is important for demonstrating domain performance. Extracted from the WDA framework, these elements form the basis for a CTA.

4.3 From Cognitive Task Analysis to Experimental Design

Cognitive Task Analysis, which is quite thoroughly described in Crandall et al. (2006), has emerged as a bridge research activity in the conceptualization of more formal experimental data collection. We design our CTA activities using information collected as we are developing the domain representations described above. In our experience, the two activities can be conducted in parallel, as long as there is continuous iterative comparision between the two. In that case, CTA and Work Domain Analysis are complementary, as the former supports validation of the domain description emerging from the latter.

In the case of the SAR imagery analysts we engaged for the PANTHER project, we developed a CTA protocol in which approximately twelve participant analysts reviewed several dozen images for particular classes of objects and signatures, similar to those they seek while on station in the SAR analysis environment. We used screen capture software to record their interactions with the imagery and developed a logging suite that captured which images the analyst was using and some of their interactions with those images (panning, zooming, switching between scenes). Once the analyst had completed the task, we immediately performed a cognitive walk through with the analyst using the video to guide the discussion, using both voice and video capture to record their subjective description of their strategy as they explained it to us.

In reviewing the data from these CTA interviews, we discovered that the SAR imagery analysts tended to rely heavily on two types of imagery made available by the SAR program’s image display software. Importantly, this finding somewhat countered the analysis process descriptions that we had documented in interviews, program documentation, and training sessions, all of which prescribed the use of additional image products for the signature detection task.

This finding played a crucial role in the development and implementation of the experimental eye tracking protocol described in the companion paper by our colleagues Matzen et al. (2015): the CTA enabled us to identify the critical, necessary-and-mostly-sufficient imagery for use in a highly abstracted task to collect data for comparing novice and expert search strategies in this domain.

5 Conclusion

Our team has used concepts from Cognitive Work Analysis and Cognitive Task Analysis to inform the collection, analysis, and representation of information that describes human activity in a complex, high-throughput work domain. We suggest that frameworks such as these help researchers balance internal and ecological validity in their experimental designs. Grounding experimental work in qualitative work domain analysis enables researchers to generate data and information that constitute valid input toward the design and evaluation of technologies intended to enhancing key elements of the perceptual and cognitive work of national security imagery analysts.