Multimodality

John Bateman, Janina Wildfeuer, Tuomo Hiippala

• Publisher: De Gruyter Mouton
• Year: 2017
• ISBN:
  9783110480047
• Edition: 1
• Topic: Languages & Linguistics
• Subtopic: Linguistics
• Index: Languages & Linguistics

This textbook provides the first foundational introduction to the practice of analysing multimodality, covering the full breadth of media and situations in which multimodality needs to be a concern. Readers learn via use cases how to approach any multimodal situation and to derive their own specifically tailored sets of methods for conducting and evaluating analyses. Extensive references and critical discussion of existing approaches from many disciplines and in each of the multimodal domains addressed are provided. The authors adopt a problem-oriented perspective throughout, showing how an appropriate foundation for understanding multimodality as a phenomenon can be used to derive strong methodological guidance for analysis as well as supporting the adoption and combination of appropriate theoretical tools. Theoretical positions found in the literature are consequently always related back to the purposes of analysis rather than being promoted as valuable in their own right. By these means the book establishes the necessary theoretical foundations to engage productively with today's increasingly complex combinations of multimodal artefacts and performances of all kinds.

Part I:Working your way into ‘multimodality’

In this part of the book we meet the basic notion of ‘multimodality’, what it is, where it occurs and which fields have addressed it. We then introduce a suitable foundation for approaching multimodality as a basic phenomenon of communicative practice.

1Introduction: the challenge of multimodality

Orientation

To get us started, we will immediately raise three related concerns: (i) what is multimodality? (ii) who is the intended audience of this book? and (iii) why might it be beneficial to know about multimodality anyway? We then give a hands-on feel for some typical cases of multimodality that have been addressed in the literature, pointing out the challenges both for the theory and practice of analysis.

If you are watching a TV news programme, where a presenter discusses some events backed up by textual overlays and recorded smartphone videos from the scene of those events, then you are interacting with a multimodal medium. If you are reading a book with diagrams and text, photographs and graphs, then you are also interacting with a multimodal medium. If you are talking to someone in a cafeteria, exchanging verbal utterances accompanied by facial expressions, gestures and variations in intonation, then you are, again, interacting in a multimodal medium. If you talk to a friend on WhatsApp and reply with an image instead of writing something, or when you draw or insert emojis on your Snapchat video, then you are, once again, communicating multimodally. And, if you are playing a computer video game (possibly in virtual reality), controlling the actions of your avatar while receiving instruction on your next mission, then this is also an example of multimodality at work. Given this, it is in fact difficult to find cases of communication and action that do not involve multimodality. So what then marks this area out as deserving a name at all? Is it not just the normal state of affairs, inherent to the nature of everyday experience in any case? Why have books on it?

Multimodality is a way of characterising communicative situations (considered very broadly) which rely upon combinations of different ‘forms’ of communication to be effective—the TV programme uses spoken language, pictures and texts; the book uses written language, pictures, diagrams, page composition and so on; talking in the cafeteria brings together spoken language with a host of bodily capabilities and postures; and the computer game might show representations of any of these things and include movement and actions as well.

Despite the fact that situations qualifying as ‘multimodal’ are everywhere, it may be surprising to discover just how little we know about how this fundamental human capability operates. We can observe that people succeed (often, but not always!) in making sense out of, and in, these multimodal communicative situations, but precisely how such varied and diverse forms of communication combine productively raises challenging issues at all levels. Moreover, the development of many traditional disciplines has focused attention in precisely the opposite direction: segmenting and compartmentalising rather than addressing how ‘ensembles’ of communicative practices work together. Thus, traditionally, linguistics deals with language, art history with paintings, etc., graphic design with page composition, architecture with buildings, film studies with film, theatre studies with theatre, and so on.

Focusing on particular areas has been immensely important for gaining deeper knowledge of the individual forms addressed. Nevertheless, as Jewitt et al. (2016) in another recent introduction to multimodality aptly put it:

“These (sub)disciplines focus on the means of meaning making that fall within their ‘remit’; they do not systematically investigate synergies between the modes that fall inside and outside that remit.” (Jewitt et al. 2016: 2)

This situation is increasingly problematic for the growing number of students, practitioners, teachers and researchers who are being confronted with the need to say something sensible, and perhaps even useful, about these complex artefacts or performances. In fact, and as we will describe in Chapter 2 in more detail, there is now barely a discipline addressing any kind of communicative situation that is not feeling the need to extend beyond the confines of the focus originally adopted.

Linguistics, for example, now seeks to take in issues of gesture in spoken language and page composition for written language; art history makes forays into addressing film; film studies moves to consider TV and other audiovisual media; and so forth. In these and many other disciplines, the awareness is growing that it is not sufficient to focus on individual ‘forms of expression’ within a communicative situation as if these forms were occurring alone. Particular forms of (re-)presentation are always accompanied by other forms: their ‘natural habitat’, so to speak, is to appear in the context of others. As a consequence, the separation of concerns that has held sway over the past 100 years or more is placed under considerable pressure.

The mechanisms of combination involved when distinct forms of expression appear together often remain obscure. And individual methods or approaches to ‘meaning making’ rarely offer the flexibility to go outside of their principal areas of focus without losing considerable analytic precision. We need then to move beyond a superficial recognition that there appears to be ‘something’ being combined and ask what it means to ‘combine’ forms of expression at a more fundamental level. Strong foundations are essential for this—otherwise the danger is either that the complexity of multimodal ‘meaning making’ will remain beyond our grasp or that only certain areas of complexity become visible at all. For this, we will need to find points of contact and overlap both in the basic functioning of combinations of expressive resources and in the methods for analysing them. In short, we need to have a better understanding of just what the ‘multi’ in ‘multimodality’ might be referring to. Just what is being combined when we talk of combinations of diverse communicative forms? It is in this area that a more focused and what we term ‘foundational’ approach is necessary.

‘Multimodality’ for the purposes of this book addresses this concern directly. It is a research orientation in its own right that seeks to address what happens when diverse communicative forms combine in the service of ‘making meanings’—however,and wherever, this is done. The claim inherent to such a position is that, when such ensembles of communicative forms appear, it is going to be possible to find similarities and parallels in the mechanisms and processes involved. This offers a complement to individual disciplinary orientations by providing more general understandings of what is going on when other forms of communication are considered. These understandings do not seek to replace existing disciplinary orientations, but rather to add ways of dealing with the particular challenges and questions that combining diverse forms of meaning-making raises.

This serves well to define the primary intended audience of this book. If you are reading this, you have probably encountered artefacts or performances that you want to study in more detail and which are made up of a variety of expressive resources. You may already have some theoretical or practical background with artefacts and performances where multimodality appears to be an issue and are interested either in taking that further, or in seeing how other approaches deal with the questions and challenges raised. What we will be developing in the chapters that follow is a framework that will allow you to pursue such investigations in a more systematic and well-founded fashion than is currently possible within individual disciplines on their own—that is: we set out a framework that places ‘multimodality’ at its core.

1.1First steps ... a multimodal turn?

All ‘multimodal’ artefacts or performances pose significant and interesting challenges for systematic investigation. It is by no means obvious just which methods, which disciplines, which frameworks can help. The range of places where issues of multimodality arise is also expanding rapidly and so it is increasingly rare that knowledge about one area will suffice. This means that our journey will turn out to be quite an interdisciplinary endeavour, although one which is held together throughout by our central orientation to the phenomenon of multimodality as such.

Multimodality in daily life

In most multimodal situations, it has already become a genuine question as to just which forms of expression might be selected. Combinations of diverse forms must now be considered to be the norm in many of the media with which we regularly engage. This is partly due to the changing media landscape both in our daily life as well as in professional environments, where we regularly face the need to communicate via several channels and with the help of various media. It is also, however, certainly due to the increasing number of products and artefacts resulting and emerging from this ever-growing landscape and our habits of using these as well. Particularly problematic for traditional divisions of analytic interest is then the fact that it will often simply not be possible to have the particular forms of expression fixed beforehand. The sheer diversity of the current media situation consequently demands that reliable, systematic, but also extensible techniques and theories for analysing such combinations be found.

As an illustration, consider Figure 1.1. Here we see a situation no doubt very familiar to everyone reading this book. These photographs were taken in a restaurant without any prior planning and so are a little rough; they show how a group of diners have formed two pairs of interactants, each for the moment pursuing their own respective concerns. There is much that is happening in these images, and so we would generally need some guidance of how to focus in on just what is to be analysed for any particular purposes we select. Broadly we see four people sitting at a table; each pair that is interacting in the picture is, however, engaged in a typical face-to-face interaction with all of the usual expressive possibilities that such interaction provides— speech, gesture, facial expression, bodily position and more.

Fig. 1.1. Everyday contemporary restaurant interactions (photographs: Oliver Kutz; used by permission)

Characterising face-to-face interactions is an area addressed by several disciplines. In some approaches complex situations of this kind are considered to be ‘layered’, or ‘laminated’ (e.g., Stukenbrock 2014), in order to pull apart the various contributions being made. We will be suggesting something similar when we turn to methods in later chapters. Moreover, in this particular case, each pair is not only engaging in face-to-face interaction, they also happen to be discussing information (of very different kinds) that is displayed on a variety of electronic devices: two smartphones in the interaction in the foreground, and a tablet and smartphone in the interaction in the background. In order to understand such interactions we then have to push the layering concept even further—we must not only deal with spoken face-to-face interaction, but also with the various information offerings and interaction possibilities present on the electronic devices (which are also multimodal!) and the ways in which the interactants communicate with each other about and with the information displayed.

We can go on and vary aspects of this situation, each variation drawing on new disciplines and expertise but actually raising the same questions again and again. Lets move, for example, the interaction from a restaurant to a study room or university cafeteria and a group of students, similarly interacting around a table with a range of technical devices, discussing their term assignment, which may involve a range of diagrams and graphs: how do we analyse how the interaction in this group works (or does not work) in order to characterise their collective activity of ‘meaning making’?

One (by now) traditional discipline would look at the spoken language interaction complete with gestures and gaze activity as the participants look at each other (as we shall mention in subsequent chapters, probably at the same time identifying ‘multimodality’ with just these particular facets of the situation). Another quite distinct community would look at the nature of diagrams, reasoning with diagrams, their construction and degrees of abstraction. The two kinds of knowledges—both essential to understanding the unfolding situation—have had virtually no interaction up until now and it is unclear how their individual descriptive frameworks would relate to one another. At the same time, another community from pedagogy and education might well have considered how people talk about diagrams and make sense of them –while drawing neither on the full range of what is known about face-to-face interaction nor on what is known about diagrams.

We can push this still further. Perhaps the students are also discussing the treatment of some controversial issue with relation to how a collection of newspapers and magazines present it (requiring knowledge of newspaper and magazine language, layout, typography, press photography), perhaps the students are discussing an artwork (requiring knowledge of art history, graphical forms and techniques), perhaps the students are not sitting around a table after all but are interacting via video (knowledge of the consequences of mediated communication for interaction), and so on.

Each of these diverse situations draws potentially on different disciplinary backgrounds, picki...

Table of contents

• Cover
• Title
• Copyright
• Contents
• How to use this book
• Part I: Working your way into ‘multimodality’
• Part II: Methods and analysis
• Part III: Use cases
• Bibliography
• Index