Chapter 1 Introduction: Shared Data in Design Research
Bo T. Christensen, Linden J. Ball & Kim Halskov
DOI: 10.1201/9781315208169-2
ABSTRACT
The Design Thinking Research Symposium (DTRS) series, of which this book is part, is an interdisciplinary symposium series linking international academics with a shared interest in design thinking and design studies coming from a diversity of disciplines, including psychology, anthropology, linguistics, philosophy, architecture, and design studies. The series provides an international forum for pioneering and state-of-the-art research on design thinking that is focused on the study of design practice from various perspectives. The 25 year history of the DTRS series is also a story of almost 25 years of shared datasets in design thinking research. This data-sharing approach was initiated in the seminal Delft Protocol Workshop (now also labelled DTRS2), which was organized by Kees Dorst, Nigel Cross and Henri Christiaans at Delft University of Technology in 1994 (Cross, Christiaans, & Dorst, 1996; Dorst, 1995) and was based around verbal protocol data collected from professional designers in a controlled context. Subsequently, two more DTRS events have involved shared data. DTRS7, organized by Janet McDonnell and Peter Lloyd, involved professional designers (architects and engineers) working in their natural habitats (Lloyd & McDonnell, 2009; McDonnell & Lloyd, 2009a, 2009b), and DTRS10, organized by Robin Adams, involved design review conversations in a design education setting (Adams & Siddiqui, 2016; Adams, Cardella, & Purzer, 2016; Adams, McMullen, & Fosmire, 2016).
At the DTRS11 25th anniversary dinner, Kees Dorst remarked in his celebratory comments that sharing design data is, first and foremost, about academic generosity. For past DTRS organizers, the labour involved in collecting and distributing the data has certainly been substantial. However, the beneficiaries of that labour are not solely restricted to the design research teams involved in the shared data analyses, but extend to the wider academic community as the receivers of the resulting publications. The three previous shared datasets have had a huge impact on the design research literature, with the resulting 3 book publications (not counting all the ensuing journal publications) attracting many hundreds of citations. But outside of design research it remains the case that shared datasets with video data are extremely rare in the humanities, social sciences and technical sciences. Nonetheless, global trends towards so-called âOpen Scienceâ clearly indicate that the sharing of video data holds substantial research potential (Adams, Radcliffe, & Fosmire, 2016).
1 Open Science
Partly spurred on by what has been dubbed the âreproducibility crisisâ in science (Baker, 2011), scholars across many disciplines have recently been pushing towards more research openness, leading to an Open Science âmanifestoâ (MunafĂł et al., 2017). Efforts to increase scientific openness aim centrally at improving the transparency of all aspects of the research process because this is viewed as being crucial to making science more reproducible whilst also bringing benefits to research efficiency. To increase the reproducibility of research results, the Open Science agenda sets great store on the value of research teams sharing data, methods and materials so as to facilitate data re-analysis and follow-on replication studies. It seems evident that past DTRS efforts involving shared datasets have already contributed as ground-breaking âfirst-moverâ cases to this Open Science agenda.
The reproducibility of research results is, however, not the only significant benefit to arise from the sharing of data such as the DTRS datasets. This is because the nature of these datasets also allows for a multitude of different research methods to be applied in their analysis. Furthermore, the fact that the data are âopen-endedâ â in the sense that their collection is not restricted to addressing a single, specific research question â allows for both inductively-oriented researchers to explore possible new theoretical angles, while simultaneously allowing for deductively-oriented researchers to test at least some theoretical design models against real-life design cases.
In this latter respect, the shared DTRS video data possess qualities resembling those emphasized as being central to design objects. Sketches, prototypes or similar design objects in-the-making are uncertain, ambiguous, re-frameable, contextually shiftable, generally open to exploration and interpretation, and basically embody qualities that provide creative potential, as captured by dominant theories of design and creativity (e.g., Dorst & Cross, 2001; Finke, Ward & Smith, 1992; Schön & Wiggins, 1992). For the individual designer such qualities allow for continual re-interpretation and object back-talk; in a collaborative setting these qualities ensure a multitude of potentially distinct perspectives being taken on the same shared object of study.
Such qualities of design objects are well-known to designers and design researchers, and shared video data of design processes can be utilized with many of the same types of benefits for the sharing parties. The shared video data thus constitute a common focal-unit of attention for all involved in the symposium, whilst also allowing for individual perspective-taking in terms of particular researchers or research groups diving into theoretically or empirically derived points of interest. In these ways the shared data facilitate discussions among the participating design researchers across the application of a variety of research methods and theoretical units of analysis. In DTRS11 moreover, such dialogue also extended to the practicing design team itself, which served not only as the object of study but also as a partner engaging in discussion and debate at the symposium.
The Open Science agenda seems to pursue mainly singular truths, by focusing narrowly on making sure that data are replicable and reproducible, which itself involves disentangling false negative and false positive results from true effects. But the variant of Open Science sought in the DTRS datasets has helped illustrate the multitude of different (typically complementary and non-competing) conclusions that may be drawn even when starting with the same data at the outset and then subjecting these data to distinct methods in order to answer a range of different research questions. In this respect, Open Science in the present volume values open-endedness in the types of research questions the data may be subjected to, in addition to pursuing the goal of allowing for reproducibility of the attained results when addressing a specific research question with either the same or different methods.
2 What to capture in shared design datasets?
In acknowledging that shared design data serve the purpose of allowing for a number of methodological approaches in the study of a range of research questions, careful consideration of what to capture seems warranted. The dataset should be rich enough to allow for individual sampling thereof for specific research purposes, and flexible enough to allow for a multitude of methodological approaches to be applied. But on the other hand, the dataset should remain manageable in order to avoid drowning in data complexities leading to analysis-paralysis, where the research teams end up spending too much time trying to gain an overview and general understanding of the content at the expense of being able to commence theoretically meaningful analyses. The dataset should also be manageable enough for the research teams to be able to finalize their research projects in time for the symposium, which is, of course, vital in order for participating researchers to utilize the collaborative potential in the shared data serving as a common ground for understanding other research perspectives. All DTRS events involving shared data have approached the question of what data to capture in a similar manner, that is, by honing in on design encounters involving verbalization in order to be able to study design activity and cognition. But the purpose of the data collection has differed greatly across symposia, as has methodological considerations on the balance between rich versus manageable datasets.
The DTRS2 Delft Protocol Workshop focused on one particular research method, verbal protocol analysis, aiming to uncover the mysterious cognitive processes involved in expert designing (Cross, Dorst, & Christiaans, 1996). The organizers collected data from five experiments involving professional designers at XeroxPARC who were working for two hours either individually or in teams of three on designing a fastening device that would allow a given backpack to be fastened onto a mountain bike. Two of these experiments were eventually shared in the dataset, involving one individual and one team-based protocol, being selected under consensual considerations of the inherent interestingness of the process and the recording quality. The lone designer worked under think-aloud instructions, where he concurrently verbalized what was going through his short-term memory while designing, whereas the design team was not given such instructions. One important observation stemming from DTRS2 concerns the disadvantages of asking for think-aloud verbalizations, notably how such instructions to enforce concurrent verbalizations may change behavior and cognitive performance (Lloyd, Lawson, & Scott, 1995; see also Davies, 1995; Schooler, Ohlsson, & Brooks, 1993, and for a contrasting po...