Introducing Multimodality

Carey Jewitt, Jeff Bezemer, Kay O'Halloran

• Publisher: Routledge
• Year: 2016
• ISBN:
  9781317269786
• Edition: 1
• Topic: Languages & Linguistics
• Subtopic: Linguistics
• Index: Languages & Linguistics

This accessible introduction to multimodality illuminates the potential of multimodal research for understanding the ways in which people communicate. Readers will become familiar with the key concepts and methods in various domains while learning how to engage critically with the notion of multimodality. The book challenges widely held assumptions about language and presents the practical steps involved in setting up a multimodal study, including:

• 
  
  

• formulating research questions

• 
  
  

• collecting research materials

• 
  
  

• assessing and developing methods of transcription

• 
  
  

• considering the ethical dimensions of multimodal research.

A self-study guide is also included, designed as an optional stand-alone resource or as the basis for a short course. With a wide range of examples, clear practical support and a glossary of terms, Introducing Multimodality is an ideal reference for undergraduate and postgraduate students in multimodality, semiotics, applied linguistics and media and communication studies. Online materials, including colour images and more links to relevant resources, are available on the companion website at www.routledge.com/cw/jewitt and the Routledge Language and Communication Portal.

Chapter 1

Navigating a diverse field

What is multimodality?

‘Multimodality’ is a term that is now widely used in the academic world. The number of publication titles featuring the term has grown exponentially since it was first coined in the mid-1990s. Since then, a myriad of conferences, monographs, edited volumes and other academic discussion forums have been produced that are dedicated to multimodality. Signs of its becoming a shorthand term for a distinct field include the publication of the first edition of the Handbook of Multimodal Analysis (Jewitt, 2009), now a revised second edition (Jewitt, 2014), the launch of the Routledge Series in Multimodality Studies (2011) and the launch of a journal titled Multimodal Communication (2012). These and many other outlets inviting contributions in the area of multimodality provide platforms for scholars working in different disciplines, including semiotics, linguistics, media studies, new literacy studies, education, sociology and psychology, addressing a wide range of different research questions.

With the term being used so frequently and widely, it may seem as though a shared phenomenon of interest has been recognized and a common object of study identified. Indeed, we can, in relatively generic terms, describe that phenomenon, or object of interest, as something like, ‘We make meaning in a variety of ways’, or, ‘We communicate in a variety of ways’. Yet we must immediately add that ‘multimodality’ (and related concepts, including ‘mode’/’modality’, ‘[semiotic] resource’) is differently construed. Exactly how the concept is articulated and ‘operationalized’ varies widely, both across and within the different disciplines and research traditions in which the term is now commonly used. Therefore, it is very difficult and potentially problematic to talk about multimodality without making explicit one’s theoretical and methodological stance.

Before going any further, we turn to those who first used the term and explore what it was that they were trying to draw attention to. As far as we can reconstruct, the term first appeared in the middle to late 1990s in different parts of the world. It is used, for instance, by Charles Goodwin, in a seminal article that he submitted to the Journal of Pragmatics in 1998 (Goodwin, 2000). It also features in Gunther Kress and Theo van Leeuwen’s Multimodal Discourse: The Modes and Media of Contemporary Communication (2001), the manuscript of which had been ‘in the making’ for a number of years. These scholars started using the term more or less independently of each other, with Goodwin in the US working in the tradition of ethnomethodology and conversation analysis, and Kress and van Leeuwen (then) in the UK in the tradition of social semiotics. Around this same time, O’Halloran, working (then) in Australia and drawing on earlier work by O’Toole (1994) and Kress and van Leeuwen (1996), began to use the term ‘multisemiotic’ to describe the multimodal character of mathematics texts (see, for instance, O’Halloran [1999b], published in Semiotica).

If a ‘means for making meaning’ is a ‘modality’, or ‘mode’, as it is usually called, then we might say that the term ‘multimodality’ was used to highlight that people use multiple means of meaning making. But that formulation alone does not accurately describe the conceptual shift these scholars were trying to mark and promote. After all, disciplines such as linguistics, semiotics and sociology have studied different forms of meaning making since well before the term ‘multimodality’ was introduced. Indeed, Ferdinand de Saussure (1857–1913), writing in the early 20th century, already suggested that ‘linguistics’ was a ‘branch’ of a more general science he called semiology. Yet the branches of that imaginary science have continued to specialize in the study of one or a small set of means for making meaning: linguistics on speech and writing, semiotics on image and film, musicology on music; and new subdisciplines have emerged: visual sociology, which is concerned with, for example, photography; visual anthropology, which is concerned with, for example, dress. These (sub)disciplines focus on the means of meaning making that fall within their ‘remit’; they do not systematically investigate synergies between the modes that fall inside and outside that remit.

Multimodality questions that a strict ‘division of labour’ among the disciplines traditionally focused on meaning making, on the grounds that in the world we’re trying to account for, different means of meaning making are not separated but almost always appear together: image with writing, speech with gesture, math symbolism with writing and so forth. It is that recognition of the need for studying how different kinds of meaning making are combined into an integrated, multimodal whole that scholars attempted to highlight when they started using the term ‘multimodality’. It was a recognition of the need to move beyond the empirical boundaries of existing disciplines and develop theories and methods that can account for the ways in which we use gesture, inscription, speech and other means together in order to produce meanings that cannot be accounted for by any of the existing disciplines. This fact only became more noticeable with the introduction of digital technologies, which enable people to combine means of making meaning that were more difficult or impossible to disseminate before – for the majority of people anyway (moving image being one pertinent example). So that is how the introduction of the notion of multimodality marks a significant turn in theorizing and analysing meaning.

What the early adopters of the term recognized was not only the need to look at the co-occurrence and interplay of different means of making meaning but also that each ‘mode’ offers distinct possibilities and constraints. It had often been argued (e.g. by Saussure and Vygotsky) that language has, ultimately, the highest ‘reach’, that it can serve the widest range of communicative functions or that it enables the highest, most complex forms of thinking and is therefore the ‘most important’. Others, including Goodwin, Kress, van Leeuwen and others who first introduced the notion of multimodality, have pointed out that there are differences between semiotic resources in terms of the possibilities they offer for making meaning but that it is not the case that one resource has more or less potential than the other. The same point was made by O’Halloran, who in her definition of ‘multisemiotic’ emphasized the significance of the combination of different resources, each with their own potential. Thus multimodality marks a departure from the traditional opposition of ‘verbal’ and ‘non-verbal’ communication, which presumes that the verbal is primary and that all other means of making meaning can be dealt with by one and the same term.

We can now formulate three key premises of multimodality:

• 1 Meaning is made with different semiotic resources, each offering distinct potentialities and limitations.
• 2 Meaning making involves the production of multimodal wholes.
• 3 If we want to study meaning, we need to attend to all semiotic resources being used to make a complete whole.

We should add four important footnotes to this. First, not everyone working in multimodality uses the notion of meaning making. Depending on their disciplinary background and focus, they might say that they are interested in ‘multimodal communication’, ‘multimodal discourse’, or ‘multimodal interaction’. We will use the term ‘meaning making’ unless we are writing about a specific approach to multimodality. Nor does everyone working in multimodality use the term ‘mode’: some prefer to talk about ‘resource’, or ‘semiotic resource’, and generally avoid drawing strong boundaries between different resources, highlighting instead the significance of the multimodal whole (‘gestalt’). Indeed, for that very reason, some scholars whose work we subsume under the heading of ‘multimodality’ do not use that term themselves, while otherwise committing to the three key premises we just presented.

Second, scholarly interest in the connections between different means of making meaning predates the notion of multimodality. For instance, the study of gesture and its relation to speech, gaze and the built environment has a long history in linguistic anthropology, interactional sociology and other disciplines (see e.g. Goffman, 1981; Kendon, 2004a; Mehan, 1980); the relation between image and writing has been studied in semiotics (e.g. Barthes, 1977 [1964]) and so on. These early contributions have produced important insights in what we now call multimodality. At the same time, we should note that the potential empirical scope of multimodality goes further still. We can see a development from an exclusive interest in language to an interest in language and its relations to other means of making meaning, to an interest in making meaning more generally, without a clear base point, whether language or any other mode.

Third, while those using the term ‘multimodality’ generally aim to develop a framework that accounts for the ways in which people combine distinctly different kinds of meaning making, their epistemological perspectives (i.e. their perspective on how we can know the world) are different. As we shall see later on in this chapter, in some approaches to multimodality the assumption is that it is possible and indeed necessary to develop an integrated theoretical and methodological framework for some kinds of meaning making, for instance for the study of speech, gesture, gaze and the material environment. In other approaches, the assumption is that it is possible and necessary to develop an encompassing theoretical and methodological framework to account for all kinds of meaning making – whether in image or in gesture or in writing or in any other mode. So researchers who adopt the notion of multimodality (or whose work is treated by others as being part of the field of multimodality) still draw different boundaries around what it is in the empirical world that they aim to account for. This is not a matter of ambition but a matter of epistemology: some argue that the differences between, say, image and speech are too great to handle within one and the same framework; others argue that, notwithstanding the differences, it is still possible, at a more general level, to establish common principles of meaning making.

Fourth, when exploring how the notion of multimodality has been and is being developed along diverse lines and schools of thought, it is important to keep an eye on the ‘original’ premises we just outlined. Fundamental to all those premises is a concern with the cultural and social resources for making meaning, not with the senses. While there are, of course, important relations to be explored between the senses and the means for making meaning, it is important not to conflate the two. The focus on the cultural and the social shaping of resources used for making meaning also sets the approaches apart from the popular notion that observation of ‘non-verbal behaviour’ offers a ‘way in’ to what an individual ‘really’ thinks (as suggested in e.g. best-selling guidebooks on ‘successful business communication’).

What makes a study ‘multimodal’?

When reviewing literature or when planning your own study, it is important to clarify what makes a study multimodal. The following sets of questions about aims, theory and method can help you assess the centrality (or marginality) of multimodality in a study:

• 1 Aims and research questions: Does it address research questions about meaning, communication, discourse or interaction? Is one aim of the study to contribute to the development of a theory of multimodality? For instance, you might find questions such as, ‘What is the semiotic relation between objects displayed in museums and their captions? What is the role of gaze in turn taking?’
• 2 Theory: What is the place of multimodality in the theoretical framework of the study? Is it a central concept, or is it referenced but not expanded on? It may also be that a theory is presented that could be described as multimodal even though it is not described as such by the authors/researchers themselves.
• 3 Method: What empirical materials are collected and analysed, and how? Do the collected materials include documentation of human artefacts and social interactions? Do the researchers attend to all (or at least a number of different) means of meaning making that can be reconstructed from the collected materials? Do they give equally systematic attention to all?

Considering the place of multimodality on these dimensions, we can distinguish between:

• Doing multimodality: Designing a study in which multimodality is central to aims/research questions, theory and method;
• Adopting multimodal concepts: Designing a study in which multimodality concepts (such as mode, semiotic resource) are used selectively.

When adopting multimodal concepts, you can draw selectively from approaches to multimodality such as the ones we discuss in the book. But picking and mixing can be a tricky approach. When selecting concepts from the frameworks and connecting them to concepts derived from other frameworks, it is important to reflect on their ‘compatibility’. Drawing on a theory raises expectations about methods used. For example, claiming to ‘use’ a theory from one of the approaches discussed in this book raises the expectation (among others, as we shall see in the next section) that you will analyse human artefacts or social interactions. So if you choose to combine that theory with the method of the interview, you are likely to be seen as having produced an incoherent framework. If you believe there are good reasons to use the interview as a method, you need to make a case for it (alternatively, you could treat the interview not as a method but as an object of study and analyse it multimodally).

Making explicit what the place of multimodality is in one’s study along these lines can be a way of setting appropriate expectations about the coherence of the research design. When you submit a research paper to a journal and suggest that the study you present is multimodal, some reviewers will expect multimodality to be central throughout the paper. When you explain that you adopt selected multimodal concepts, reviewers are more likely to assess the ‘fit’ between those concepts and the theoretical and methodological frame within which you integrate it. We will elaborate on the issue of mixing approaches in Chapter 6.

Three approaches to multimodal research

In Chapters 3, 4 ...

Table of contents

• Cover
• Title
• Copyright
• Contents
• List of illustrations
• About this book
• Acknowledgements
• Chapter 1 Navigating a diverse field
• Chapter 2 Why engage with multimodality?
• Chapter 3 Systemic functional linguistics
• Chapter 4 Social semiotics
• Chapter 5 Conversation analysis
• Chapter 6 Five more approaches to multimodality
• Chapter 7 Designing a multimodal study
• Glossary
• Self-study guide
• References
• Index