The Structure of Multimodal Documents

An Empirical Approach

Tuomo Hiippala

• Publisher: Routledge
• Year: 2015
• Print ISBN:
  9781138548725
• eBook ISBN:
  9781317580126
• Topic: Languages & Linguistics
• Subtopic: Communication Studies
• Index: Languages & Linguistics

This book develops a new framework for describing the structure of multimodal documents: how language, image, layout and other modes of communication work together to convey meaning. Building on recent research in multimodal analysis, functional linguistics and information design, the book examines the textual, visual, and spatial aspects of page-based multimodal documents and employs an analytical model to describe and interpret their structure using the concepts of semiotic modes, medium and genre. To demonstrate and test this approach, the study performs a systematic, longitudinal analysis of a corpus of multimodal documents within a single genre: an extensively annotated corpus of tourist brochures produced between 1967-2008. The book provides multimodal discourse analysts with methodological tools to draw empirically-based conclusions about multimodal documents, and will be a valuable resource for researchers planning to develop and study multimodal corpora.

Routledge Studies in Multimodality

Edited by Kay L. O’Halloran, Curtin University

1. New Perspectives on Narrative and Multimodality
   Edited by Ruth Page
2. Multimodal Studies
   Exploring Issues and Domains
   Edited by Kay L. O’Halloran and Bradley A. Smith
3. Multimodality, Cognition, and Experimental Literature
   Alison Gibbons
4. Multimodality in Practice
   Investigating Theory-in-practice-through-methodology
   Edited by Sigrid Norris
5. Multimodal Film Analysis
   How Films Mean
   John A. Bateman and Karl-Heinrich Schmidt
6. Multimodality and Social Semiosis
   Communication, Meaning-Making, and Learning in the Work of Gunther Kress
   Edited by Margit Böck and Norbert Pachler
7. Spoken and Written Discourse in Online Interactions
   A Multimodal Approach
   Maria Grazia Sindoni
8. Film Discourse Interpretation
   Towards a New Paradigm for Multimodal Film Analysis
   Janina Wildfeuer
9. Critical Multimodal Studies of Popular Discourse
   Edited by Emilia Djonov and Sumin Zhao
10. Multimodal Approaches to Research and Pedagogy
    Recognition, Resources, and Access
    Edited by Arlene Archer and Denise Newfield
11. Multimodal Epistemologies
    Towards an Integrated Framework
    Edited by Arianna Maiorani and Christine Christie
12. Multimodal Analysis in Academic Settings
    From Research to Teaching
    Edited by Belinda Crawford Camiciottoli and Inmaculada Fortanet-Gómez
13. The Structure of Multimodal Documents
    An Empirical Approach
    Tuomo Hiippala

The Structure of Multimodal Documents

An Empirical Approach

Tuomo Hiippala

First published 2016

by Routledge

605 Third Avenue, New York, NY 10017

Simultaneously published in the UK

by Routledge

2 Park Square, Milton Park, Abingdon, Oxon OX14 4RN

Routledge is an imprint of the Taylor & Francis Group, an informa business

© 2016 Taylor & Francis

The right of Tuomo Hiippala to be identified as author of this work has been asserted by him in accordance with sections 77 and 78 of the Copyright, Designs and Patents Act 1988.

All rights reserved. No part of this book may be reprinted or reproduced or utilized in any form or by any electronic, mechanical, or other means, now known or hereafter invented, including photocopying and recording, or in any information storage or retrieval system, without permission in writing from the publishers.

Trademark Notice: Product or corporate names may be trademarks or registered trademarks, and are used only for identification and explanation without intent to infringe.

Library of Congress Cataloging-in-Publication Data

Hiippala, Tuomo, 1982– author

The structure of multimodal documents : an empirical approach / Tuomo

Hiippala.

pages cm. – (Routledge studies in multimodality ; 13)

Includes bibliographical references and index.

ISBN 978-1-138-82474-4 (hardback) – ISBN 978-1-315-74045-4 (ebook)

1. Written communication. 2. Modality (Linguistics) 3. Linguistic analysis

(Linguistics) 4. Structural linguistics. I. Title.

P211.H57 2015

302.2’244–dc23

2015017871

ISBN: 978-1-138-82474-4 (hbk)

ISBN: 978-1-315-74045-4 (ebk)

DOI: 10.4324/9781315740454

Typeset in Palatino

by Tuomo Hiippala

Contents

• 1 Introduction
  • 1.1 Why use linguistic methods for multimodal research?
  • 1.2 Why do empirical research on multimodality?
  • 1.3 Why focus on multimodal structure?
  • 1.4 Why study the tourist brochures?
  • 1.5 Organisation of the book
• 2 Understanding page-based media
  • 2.1 The state of the art
  • 2.2 Turning back the page
  • 2.3 Defining a mode
  • 2.4 Production and consumption
• 3 An empirical approach to multimodality
  • 3.1 The Genre and Multimodality (GeM) model
  • 3.2 The analytical layers of the GeM model
  • 3.3 The concept of a semiotic mode
  • 3.4 Semiotic modes in the tourist brochures
  • 3.5 Page-flow in the tourist brochures
• 4 Genre: perspectives and patterns
  • 4.1 An outlook on deploying genre
  • 4.2 The study of rhetoric: genre as social action
  • 4.3 Linguistic studies of genre structure
  • 4.4 Genre dynamics
  • 4.5 Genre in document theory and information design
  • 4.6 Describing the structure of a multimodal artefact
• 5 Working with multimodal corpora
  • 5.1 Selecting the data
  • 5.2 Applying the GeM model
  • 5.3 Building the corpus
  • 5.4 Verifying the corpus
  • 5.5 Exploring the corpus
• 6 The medium and its characteristics
  • 6.1 Why the notion of medium matte...

Table of contents

• Cover Page
• Half Title Page
• Frontmatter 1
• Title Page
• Copyright Page
• Table of Contents
• 1 Introduction
• 2 Understanding page-based media
• 3 An empirical approach to multimodality
• 4 Genre: perspectives and patterns
• 5 Working with multimodal corpora
• 6 The medium and its characteristics
• 7 The content and its structure
• 8 The page and its interpretation
• 9 Multimodal artefacts in digital media
• 10 Conclusions and outlook
• A The entire data set
• B The rhetorical relations used in GeM RST
• Bibliography
• Index