Interactions, Images and Texts

Sigrid Norris, Carmen Daniela Maier

• Publisher: De Gruyter Mouton
• Year: 2014
• ISBN:
  9781614518136
• Edition: 1
• Topic: Languages & Linguistics
• Subtopic: Linguistics
• Index: Languages & Linguistics

Multimodality is a fast-growing interdisciplinary approach that aims to analyze the interplay of multiple modes such as gaze, gesture or spoken language that are utilized in interaction, and to examine the multimodal production and consumption of communicated messages.
This Reader provides a comprehensive text of current research into multimodality, outlining in-depth delineation of each primary theoretical and methodological approach, as well as personal accounts of scholars, who are responsible for the various approaches' advancements. The book additionally offers a plethora of analysis chapters, written by scholars from across the world, with vastly diverse themes ranging from buying popcorn, protests in Oman, coaching sessions and identity, to kitesurfing, typography, TV news, billboards, workplace practices, or analyzing web pages, Facebook, comic books, and more.
Flexible and easy to use, the Reader includes key terms, suggested further readings, and a project idea for each chapter. The key terms for the chapters also comprise the extensive alphabetical glossary. Brief introductions for the analysis chapters, written by the editors, summarize the topic, explain the methodology used, outline the thematic orientation, and link each chapter to other chapters in the book.
Showcasing multimodal analysis in detail, this Reader is essential reading for undergraduate and postgraduate students, for emergent researchers, and for advanced scholars who wish to gain insight into the current state of multimodal research.

Sigrid Norris and Carmen Daniela Maier

1 Introduction

We are seeing a fast increase of interest in multimodal communication in various departments from business and design, communication studies and linguistics, psychology and sociology to health. This rise of interest also brings with it an interest in teaching multimodality in the undergraduate and the graduate classroom.

At the 2010 international conference titled Multimodality, Mediation and Practice held by the Multimodal Research Centre at Auckland University of Technology in New Zealand, scholars from a variety of approaches to multimodality attended. Due to the breadth and depth of presentations, a lively discussion about the emerging field of multimodality resulted and a number of lecturers and researchers lamented the fact that there is no Reader that can trace and encompass the increasing variety of approaches.

Discussing the need for such a book further after the conference, we decided to write/edit one that would give a taste of the breadth of theoretical, methodological and analytical issues in the area of multimodality. Thus, Interactions, Images and Texts: A Reader in Multimodality was conceived. It has taken some time to bring together all of the various authors and to actually put this Reader together, but we hope that it will prove useful for the undergraduate and postgraduate classroom, as well as to emergent researchers and researchers new to the field of multimodality. You will find the book equipped with key terms for each chapter that are defined in the alphabetised glossary for a quick reference. The terms were listed and defined by the particular authors and the names of the ones who defined them are listed in the glossary. We have decided to explicate exactly who defined a particular term as we find more and more examples in the literature where the seemingly same terms are used differently by different authors. This way, we hope to have alleviated some terminology confusion and enable the reader to understand the terms as the author(s) intended.

There is a short list of further readings at the end of each chapter. These readings are suggested by the author(s) who wrote the particular chapter, and are therefore, very useful suggestions when you become interested in reading more about specific topics. After these reading suggestions, each author has provided the reader with a project idea. These project ideas allow the book to be easily used as a teaching tool or as a guide for self-study. With these further readings, possible projects and an alphabetic glossary at the end of the book, we hope to have made the book useful for the various levels.

dp n="16" folio="2" ?

We have structured the book into four sections: I. Multimodal theory and methodology: How are they developed? Here, the reader will find personal accounts of some leading figures in the field of multimodality. These chapters are written in different styles, recounting some aspects of how, when, or why these theories or methodologies came about. We believe that it will be an interesting and varied read, where the reader will gain an insight into these authors’ personal stories. In this section, the readers can discover the personal accounts of Suzie Wong Scollon about developing mediated discourse and nexus analysis, of Sigrid Norris about developing multimodal (inter)action analysis, of Theo Van Leeuwen about developing social semiotics, and of John Bateman about developing multimodal genre analysis.

In section II, Multimodal theory and methodology: What are their facets? we offer theoretically and methodologically focused chapters using and/or explaining the theories/methodologies developed by the scholars from the first section. These chapters are structured in the way discussed above, with key terms listed after the introduction of each chapter, further readings following the conclusion, and project ideas following the further readings. In this section, Rodney Jones explains and exemplifies in detail how mediated discourse analysis can be performed. Nexus analysis is the focus of Yuling Pan’s chapter, while Najma Al Zydjaly introduces the main aspects of geosemiotics, and Jesse Pirini addresses multimodal (inter)action analysis. Carmen Daniela Maier investigates a new genre of corporate videos using a social semiotic approach, while Tuomo Hiippala closes this section with a multimodal genre analysis using the GEM model.

In section III, Conducting multimodal research, we present chapters that go into more depth in how-to do multimodal analyses and that illustrate the practical side of multimodal research. This section opens with Carey Jewitt’s chapter on aspects of multimodal research. Kay O’Halloran and Victor Lim Fei elaborate in their chapter on the systemic functional approach, presenting in detail the primary methodological tools needed for this type of analysis. Acknowledging the challenges of multimodal transcription, Jeff Bezemer concentrates on explaining how he dealt with transcription in his case study. How to analyse webpages is the question posed by John Knox and Emilia Djonov in their chapter. Rick Iedema familiarizes the reader with a participatory approach to sense-making of visual data, illustrating the consequences of involving practitioners in video-based research. This section of the book closes with two chapters co-written by Sigrid Norris, Jarret Geenen, Thomas Metten, Tui Matelau and Jesse Pirini who investigate the role of the researcher and aspects of ethics when collecting video data.

Then, in section IV, Sample analyses, we showcase smaller analysis chapters that give the reader a good understanding of the breadth of uses of multimodal research methodologies. Jarret Geenen offers an analysis of extreme sports, Tui Matelau analyses Maori woman identity, while Arianna Moriani provides the reader with an example for the study of film advertising posters in her chapter. Hartmut Stöckl presents specific guiding principles needed for the multimodal analysis of typography. Sabine Tan turns to the analysis of news, while Volker Eisenlauer approaches Facebook, explaining the disempowering effects of the standardized options of the online environment upon its users. The communicative aspects of space in action are described by Paul White, while Gill Abousnnouga and David Machin focus on monuments. Ingrid de Saint-Georges addresses mediated discourse analysis referring to embodied learning and emerging social and professional identities. In Maria Jesus Pinar’s chapter, the focus is on comic books, while Alison Gibbons elaborates on aspects of multimodality in literature in the section’s last chapter.

In this section, we, the editors, introduce each chapter briefly, giving the reader a quick overview of the thematic orientation, the theoretical or methodological orientation, and the linkages to other similar chapters. Here, too, we provide key terms, further readings and a project idea.

The book can be read in various ways so that you can, as with any book begin to read from the first to the last section. But you may also, for example, begin working with one analysis chapter, find the links to other similar chapters and then move from there to the linked methodology in section two or the practical chapters in section three and read the personal accounts of those authors whose framework you are most interested in. Thus you may well be skipping back and forth between sections. We structured the book in this way on purpose, as we wanted this Reader to be a useful book that is adoptable by as many kinds of ways of teaching/learning as possible; allowing the teacher/student to decide which way is best for them.

As mentioned in the beginning, the idea and the content of this book were inspired by the insights that we gained during our discussions at a conference with other fellow researchers interested in teaching and working with various aspects of multimodality. It is our hope that this Reader will stimulate more discussions at conferences as well as in journals and classrooms across disciplines and geographical borders, driving multimodality research forward.

I Multimodal theory and methodology: How are they developed?

dp n="20" folio="6" ? dp n="21" folio="7" ?

Suzie Wong Scollon

2 From mediated discourse and nexus analysis to geosemiotics: A personal account

Introduction

Mediated discourse analysis emerged at the turn of the millennium as Ron Scollon and colleagues combined discourse analysis with mediated action in an effort to develop activist sociolinguistics, to analyze discourse mediated by different means with an eye to effective social action. Ron returned to his dissertation done in the 70s to develop an ontogentic view of social practice, linking a one year-old’s handing and speech in a nexus of practice (Scollon 2001). MDA has been used to examine such issues as immigration, unemployment, genetically modified corn, anthropogenic climate change and tar sands extraction and transport. Ron asked what an actor was doing, and how their speech/action arose out of a nexus of practice including narrative, first spoken then printed.

Narrative in MDA

Ron began paying attention to narratives in the early 70s as he developed mediated discourse analysis. In December 1972 we witnessed actions leading to escalation of the Vietnam war with a massive bombing of Hanoi and Haiphong in the secret “Operation Linebacker II”. Though everyone in Honolulu could have observed the escalation of heavy tankers flying low to fuel B52 bombers in Guam, no protest action was possible because protest leaders became agents only in response to what they read in print, and there was a delay before an account appeared in I.F. Stone’s Weekly, which made Ron’s report a vain endeavor.

This early work in narrative fell under the rubric of the ethnography of communication, as he thought through ways in which stories were told in relation to participants and events. For example, when he was visiting my cousin Brenda in order to record her speech development, Brenda’s mother told him a story and repeated it with embellishment when I appeared. He hypothesized that she did that to retain his interest as he had already heard the basic story. He looked at motives after Burke and continued to look at narratives as explanatory framings of self, role, etc. after Goffman, asking how and why narratives were told. He noted a kind of magnetism, with himself as analyst getting lost in the actions of the tales he was analyzing.

After our daughter was born, Ron recorded our speech around her (e.g. when she fell down and he tried to suggest a narrative to transform Rachel’s crying into speech). Did she trip? I suggested an alternative explanation. Perhaps she was tired or dizzy. He told me a long narrative over coffee after breakfast, displaying his current work on Athabaskan, the only long narrative all day while Rachel was awake and the recorder was running. Did it inform me? Was it intended to? It was a habit developed before Rachel was born, as we walked the four miles from our downtown apartment to the university campus where he discussed Brenda’s speech at a child language seminar.

The question Ron asked was: What makes a story a narrative? He noted a tendency to hear all past tenses or perfectives as narrative, and past tense transitive clauses as narrative clauses. He found the uses of narrative changing, with narrative becoming a style of telling that relies on temporally sequencing events whether real, imaginary, or mental, a kind of word form object or genre. He read Uspensky, who had been at Berkeley at the same time as Goffman. An associate of Bakhtin, Uspensky wrote on point of view in citation.

Looking at narrative use in newspapers, Ron found himself mostly just following stories, so he decided to trace a story from its first appearance on the radio news through newspaper and TV, comparing versions against each other. The bombing of LaGuardia airport at the end of 1975 captured his attention for days during the winter semester break. He noted:

The bomb explosion itself caused a change of frame, a group of frame breaks.... The story is

1. we were sitting around waiting
2. the explosion happened
3. we were casualties

He listened to the news on another radio station at 8am. The Dec. 31 Advertiser had a “screaming red headline” with 9 paragraphs adding only that the airport is open again and a reward has been announced. It was this headline that caught his eye as he walked past a news stand that holiday morning.

Questioning Labov’s definition of narrative as recapitulating an event, he writes, “The event exists only as an interpretation of physical and behavioral phenomena.” ... “Activities do not become events until organized and that organization is an embedding. The structure is the embedding.”

Nexus analysis in Alaska

Nexus analysis as a tool for unifying micro-analysis of social interaction and broader socio-political-cultural analysis started in Alaska more than three decades ago as Ron and I conducted projects directed toward improving the access of Alaska Native people to public institutions from which they were being systematically excluded largely because of communicative technologies and practices. The institutions were educational, medical, legal, and economic.

An early activity was a booklet ‘How to do beadwork’ for children in a bilingual education program that led us to an analysis of text, images and interactions. The first page was a picture of a standing moose with text in Ahtna Athabaskan and English translation below. The last page showed a pair of feet covered by beautifully beaded moccasin tongues (Scollon and Scollon 2004). What the authors showed was the context, the reasons for doing beadwork, rather than the beadwork itself. Children would learn how to do that by watching their mothers.

The bilingual education program for Ahtna children was part of a nexus of legal pressures for such services mandated by the US Supreme Court in Lau vs. Nichols in 1974. Nothing happens in a social vacuum but rather in a nexus of historical trajectories of people, places, discourses including texts, images and interactions, ideas, and objects. In 1968 oil was discovered at Prudhoe Bay on the Arctic coast of Alaska. The Natives of the state sued the US government claiming ownership of all the land in the state, which resulted in a cash settlement, title to large portions of land, and health and social services. The world oil trade and the Alaska Native Claims Settlement Act formed two cycles in the nexus we were involved in. Both were subsumed by the world system then dominated by Cold War tensions between the US and the Soviet Union.

While Ron flew around the state, which has an area of 1,500,000 square kilometers, consulting and giving workshops to Alaska Natives and legal, educational and medical professionals, I taught Alaska Native students who wanted to become teachers while living at home instead of moving to Fairbanks where the University of Alaska is located. They were scattered across the state, 800 miles from east to west and 900 miles from north to south. I tried tape recorded audio lectures, videotapes, telephone conferencing and mail correspondence before stumbling on computer mediated communication. From earlier research on interaction between Athabaskans and ourselves as well as other outsiders, we posited that the asynchronous nature of computer communication would facilitate the teaching and learning of students who spoke or heard their elders speaking Athabaskan languages. It turned out that Athabaskan and other students who did not easily speak in class or visit professors’ offices used this medium readily.

Whether discourse was mediated by computer terminals, telephones and microphones, or books and printed paper made a difference to people distantly related to an elder who refused to be tape recorded saying, “Separating the word from the body is death.” We began to pay attention to mediation by machines as well as Goffman’s interaction order as we noted that Athabaskans preferred narrative to conversation, and traced these interactive preferences in their historical bodies.

Geosemiotics

The landscape changed as we went from being students and impoverished researchers in Honolulu to faculty in a state newly enriched by oil. Instead of pedaling our bicycles in the mainstream traffic to get to classes, we flew and drove around a state more than twice the size of Texas, the next largest state. Not only were we navigating small icy landing strips and snow-covered roads, bicycle lanes appeared in Fairbanks as well as Honolulu as we tried to read the signs and listen to the sounds of Alaska Native languages.

Our first project in Alaska, before moving the family there, was to find a one year-old to compare with the subject of Ron’s PhD dissertation. Unlike Brenda’s milieu with books, TV, automobiles, shopping malls and street signs, the community where we located ourselves had only one gravel road from the airport to the village and one small store which sold no books. People spent more time on boats on the river than on wheels, and signs were not lettered but signs of wear where branches were broken and bushes trampled on.

That was the beginning of widening the circumference of study in time and space as we tried to account for differences in functions of language. We gathered blueberries, hunted caribou and heard more buzzing mosquitoes than words of Gwich’in. I learned to sew beads and watched women tan moosehide. The ecology of language in Arctic Village in the Brooks Range could not have been more different than that in Pearl City, Oahu where Brenda lived with her family. The discourses, material and ephemeral, were different, the historical bodies were different, and the interaction orders were different. We did not hear mothers speaking to babies or reading to toddlers. The only reading we observed took place at the post office and at church.

Signs as multimodal objects

Twenty years later we found ourselves in yet another ecology, that of ultra-urban Hong Kong, where forests of signs hang out over streets. Though there was a plethora of bilingual signs, we discovered that our students pretty much ignored the English portions just as we ignored the Chinese unless we forced ourselves to pay attention. We undertook an ethnography of communication centering on newspaper texts and readers, and marveled at how our students spoke to each other in Cantonese while creating English language texts.

We often traveled to China, where we heard Chinese languages that were not Cantonese and saw signs posted in simplified characters. I picked up a book about different fonts, and we began photographing street and shop signs, analyzing the placement of these multimodal objects in relation to the character sets, fonts, colors and other material characteristics such as whether they were to be read from left to right or from right to left.

We found to our surprise that our students were at least as disoriented in a city just a short distance from Hong Kong as we were. Though they spoke Cantonese, the language of Guangzhou, due to differences in their historical bodies, interaction orders and media use they felt lost. Their mobile phones did not work and they were accustomed to tracking...

Table of contents

• Trends in Applied Linguistics
• Title Page
• Copyright Page
• Table of Contents
• Contributors
• 1 Introduction
• I Multimodal theory and methodology: How are they developed?
• II Multimodal theory and methodology: What are their facets?
• III Conducting multimodal research
• IV Sample analyses
• Glossary
• Index