Analyzing Multimodal Interaction

A Methodological Framework

Sigrid Norris

• Publisher: Routledge
• Year: 2004
• ISBN:
  9781134333868
• Edition: 1
• Topic: Languages & Linguistics
• Subtopic: Linguistics
• Index: Languages & Linguistics

Our perception of our everyday interactions is shaped by more than what is said. From coffee with friends to interviews, meetings with colleagues and conversations with strangers, we draw on both verbal and non-verbal behaviour to judge and consider our experiences.
Analyzing Multimodal Interaction is a practical guide to understanding and investigating the multiple modes of communication, and provides an essential guide for those undertaking field work in a range of disciplines, including linguistics, sociology, education, anthropology and psychology. The book offers a clear methodology to help the reader carry out their own integrative analysis, equipping them with the tools they need to analyze a situation from different points of view. Drawing on research into conversational analysis and non-verbal behaviour such as body movement and gaze, it also considers the role of the material world in our interactions, exploring how we use space and objects - such as our furniture and clothes - to express ourselves. Considering a range of real examples, such as traffic police officers at work, doctor-patient meetings, teachers and students, and friends reading magazines together, the book offers lively demonstrations of multimodal discourse at work.
Illustrated throughout and featuring a mini-glossary in each chapter, further reading, and advice on practical issues such as making transcriptions and video and audio recordings, this practical guide is an essential resource for anyone interested in the multiple modes of human interaction.

1 Multimodal interaction

All interactions are multimodal. Imagine, for example, a simple two-person interaction, a conversation with a friend. During this interaction, you are aware of your friend’s spoken language, so that you hear the verbal choices, the content, the prosody, and the pitch. You are also aware of the way that your friend is standing or sitting, the way that your friend is nodding or leaning back or forward; you are aware of your friend’s facial expression, and clothing, just as you are aware of the environment in which this interaction takes place. If there is music playing in the background, even though you are not focusing on the music, you are aware of it. All of these elements play a part in this conversation. You may react to the words that your friend is speaking as much as you may react to your friend’s facial expression or the posture that your friend is taking up towards you. You may speak quickly or slowly, depending on the music playing in the background or the given environment that the interaction takes place in.

Intuitively we know that we draw on all of these communicative channels or modes when interacting with others. We also know that we are aware of many things that surround us while we interact with others. Let us keep thinking about a conversation. No matter where it may take place, you are certainly aware whether other people are present in close proximity. Thus, if your conversation takes place at a table in a cafeteria, you are aware of others talking, eating, or passing by your table. You may not take much notice of these other people, because you are focused on your conversation, but you are aware of them nevertheless.

Interactional meaning

Generally it is assumed that we can communicate best through our use of language. Language seems to have the most informative content, which can easily be employed without a need for other channels. We may speak on the phone, write emails, or go to chat-rooms. In each case, we use language, either spoken or written, to communicate.

But when thinking about TV or the Internet, it is clear that we also communicate through images. Often, viewing an image may carry more communicative meaning than reading a description of the very same thing. We may even feel that the image has more “reality” to it than a written description of the same image would have. This realization questions the notion that the process of communicating is dependent upon language.

Just as moving images or still photos can communicate meaning to the viewer, nonverbal channels such as gesture, posture, or the distance between people can — and do — carry meaning in any face-to-face interaction. All movements, all noises, and all material objects carry interactional meaning as soon as they are perceived by a person.

Previously, language has been viewed as constituting the central channel in interaction, and nonverbal channels have been viewed as being subordinated to it. While much valuable work on the interplay between the verbal and nonverbal has been established, I believe that the view which unquestionably positions language at the center limits our understanding of the complexity of interaction.

Therefore, I will step away from the notion that language always plays the central role in interaction, without denying that it often does. Language, as Kress et al. (2001) have noted, is only one mode among many, which may or may not take a central role at any given moment in an interaction. In this view, gesture, gaze, or head movement may be subordinated to the verbal exchanges going on as has been shown in much research. However, gesture, gaze, and head movement also may take the superior position in a given interaction, while language may be subordinated or absent altogether. Alternatively, sometimes many communicative channels play an integral part in a given interaction, without one channel being more important than another.

While we all intuitively know that people in interaction draw on a multiplicity of communicative modes, and that people in interaction are aware of much more than just what they are focused upon, an analysis of such multimodal interaction brings with it many challenges.

Structure and materiality

One challenge for the analysis of multimodal interaction is that the different communicative modes of language, gesture, gaze, and material objects are structured in significantly different ways. While spoken language is sequentially structured, gesture is globally synthetically structured, which means that we cannot simply add one gesture on to another gesture to make a more complex one. In language, we can add a prefix to a word, making the word more complex; or we can add subordinate clauses to a main clause, making the sentence more complex. With gestures, this is not possible, since gestures that are linked to language inform about global content or intensity. Gaze, however, may be sequentially structured, and during conversation it often is. But, during other interactions, gaze can be quite random. For example, when you walk through the woods with a friend, your gaze may wander randomly, focusing on a tree, a rock, or nothing at all. Then there are other communicative modes, which are structured even more differently. As we will see, furniture is a mode, and when thinking about it, we find a functional structure. Chairs are usually located around a table, or a reading lamp is located next to an easy chair. Thus, different modes of communication are structured in very different ways.

Another challenge for the analysis of multimodal interaction is the fact that different communicative modes possess different materiality. For example, spoken language is neither visible nor enduring, but it does have audible materiality. Gesture, however, has visible materiality but is also quite fleeting. The mode of print has more visible materiality and is also enduring; and the mode of layout, thinking about furniture, for example, has highly visible materiality and is extensively enduring.

Awareness and attention

Our next challenge for the analysis of multimodal interaction is the need to explicitly link the analysis of interaction to an analysis of a person’s awareness.

When conducting multimodal analysis of human interaction, we need to consider the human mind. I am partial to the notion of a duality of mind, as discussed in great detail by Chalmers (1996). He discusses the theoretical division of mind as the following two concepts: first, the psychological concept, which is the process whereby cognitive systems are sensitive to outside stimulation and result in a psychological change. This is the part of the mind in which people experience perceptions, thoughts, and feelings, but “it matters little whether a mental state has a conscious quality or not. What matters is the role it plays in a cognitive economy” (Chalmers, 1996: 11). Second, Chalmers speaks of the phenomenal concept of mind. This is the concept of mind as conscious experience or simply the state of awareness. While both concepts of mind often if not always co-occur, this division allows us to focus on just one part of consciousness.

Chalmers proposes that:

A good test for whether a mental notion M is primarily psychological is to ask oneself: Could something be an instance of M without any particular associated phenomenal quality? If so, then M is likely psychological. If not, then M is phenomenal, or at least a combined notion that centrally involves phenomenology.

(Chalmers 1996:18)

In interactional multimodal analysis, we are not much concerned with the perceptions, thoughts, and feelings that people are experiencing, but we are concerned with the perceptions, thoughts, and feelings that people are expressing. We can surmise that some perceptions, thoughts, and feelings that are expressed by someone are also somewhat experienced by that person, even though the actual experience and the expression of the experience should not be viewed as a one-to-one representation and may be as diverse as to contradict each other. We can also surmise that not every perception, thought, and feeling that a person experiences is expressed.

A part of conscious experience, namely interactional awareness, can be analyzed qualitatively. In order to do this, we analyze not only the messages that an individual in interaction sends, but also how other individuals in the interaction react to these messages. Thus, there is a constant tension between what a person consciously does and what that person expresses. Interaction, then, is the exchange of communicated (expressed, perceived, and thereby interpreted) experiences, thoughts, and feelings of participants.

Let us return to our example in a cafeteria, and imagine that there is a big window to the right of the table. When you gaze out of the window during the conversation, your friend may interpret this action as a sign of either boredom or deep concentration, or may not take any notice of it. Only through an analysis of you and your friends’ actions can an analyst come to a conclusion about the communicative meaning that this action of looking out of the window has in this interaction.

In multimodal interactional analysis we are only concerned with what individuals express and others react to. We are not concerned with what people are actually perceiving, thinking, and feeling, which may sometimes be different and more complex than what they express. We are also not concerned with a person’s intentionality, which sometimes may be different from what is expressed or different from what is perceived by others.

Multimodal interactional analysts set out to understand and describe what is going on in a given interaction. We analyze what individuals express and react to in specific situations, in which the ongoing interaction is always co-constructed. Furthermore, it is of utmost importance to realize that one and the same action — like looking out of a window — can have many different meanings, intentional or unintentional.

While we react to the words, gestures, facial expressions, etc. of others, we are rarely interested in what others are thinking. For example, when you visit a retail store to buy a present for your mother, and you talk to the sales person, you are not interested in what the sales person is thinking, but whether or not the sales person is showing you items that your mother would like. Considering what others are thinking is rare and usually comes about only in intimate situations.

Yet, what we do consider is an aspect of the phenomenal concept of mind. We consider whether a person is kind to us or shows anger, whether a person we interact with reacts in an appropriate manner or not, and whether or not somebody is paying attention to us. As humans, we are excellent at noticing whether others are paying attention, and we learn from early childhood onwards that we are supposed to realize whether we can approach another human being or whether the other person is occupied.

Thus, consciousness for the multimodal interactional analyst — just as for any participant in an interaction — means the analysis of another social actor’s awareness. In the Dictionary of Psychology, we find the following definition of awareness:

awareness: 1. consciousness; alertness; 2. cognizance of something; a state of knowledge or understanding of environmental or internal events.

(Chaplin, 1985; emphasis is my own)

Awareness and attention can be used somewhat interchangeably, even though there are slight differences in their meaning. When looking at the Dictionary of Psychology again, we find attention to be defined as follows:

attention: 1. the process of preferentially responding to a stimulus or range of stimuli; 2. the adjustment of the sense organs and central nervous system for maximal stimulation; 3. (Titchener) a state of sensory clearness with a margin and a focus.

(Chaplin, 1985)

When you think about the definitions, you will find that you are certainly aware of something that you are paying attention to, and you also pay attention to something that you are aware of. But, awareness/attention comes in degrees. Let me refer once more to the Dictionary of Psychology:

attention level: the degree of clarity of an experience ranging from unconsciousness (total lack of awareness) to focal attention (vivid awareness).

(Chaplin, 1985; emphasis is mine)

Consequently, when analyzing human interaction, we are concerned with only two aspects of the phenomenal concept of mind:

1. with the expressions of perceptions, thoughts, and feelings; and
2. with the different attention/awareness levels.

Certainly, these two aspects are combined in interaction, as expressions of perceptions, thoughts, and feelings are essentially communicated on different levels of attention/awareness. All participants in a given interaction react to the expressed aspect of the phenomenal mind of others, and it is this aspect of mind that is observable by the analyst.

Chalmers poin...

Table of contents

• Cover Page
• Title Page
• Copyright Page
• List of Figures
• List of Plates
• Preface
• Acknowledgments
• 1 Multimodal interaction
• 2 Communicative modes
• 3 Multimodal transcription
• 4 Modal density
• 5 Levels ofattention/ awareness
• 6 Semantic/pragmatic means
• 7 Modal density foreground– background continuum as a methodological framework
• 8 Analyzing multimodal interaction
• References