Text and Image

A Critical Introduction to the Visual/Verbal Divide

John Bateman

• Publisher: Routledge
• Year: 2014
• ISBN:
  9781317683025
• Edition: 1
• Topic: Languages & Linguistics
• Subtopic: Linguistics
• Index: Languages & Linguistics

Text and image are used together in an increasingly flexible fashion and many disciplines and areas of study are now attempting to understand how these combinations work.This introductory textbook explores and analyses the various approaches to multimodality and offers a broad, interdisciplinary survey of all aspects of the text-image relation. It leads students into detailed discussion concerning a number of approaches that are used. It also brings out their strengths and weaknesses using illustrative example analyses and raises explicit research questions to reinforce learning.

Throughout the book, John Bateman looks at a wide range of perspectives: socio-semiotics, visual communication, psycholinguistic approaches to discourse, rhetorical approaches to advertising and visual persuasion, and cognitive metaphor theory. Applications of the styles of analyses presented are discussed for a variety of materials, including advertisements, picture books, comics and textbooks.

Requiring no prior knowledge of the area, this is an accessible text for all students studying text and image or multimodality within English Language and Linguistics, Media and Communication Studies, Visual and Design Studies.

PART I Relating Text and Image

In this first part of the book we begin by considering a little more carefully just what might be meant by ‘text’ and ‘image’ – we will see that this is by no means as clear as might have been thought. We also see just why a consideration of how text and image work together is both scientifically interesting and of considerable practical import. Given that text and image now appear together in increasingly flexible ways, the interpretative demands raised for readers and viewers are also growing. Unit 1 gives us our first view of the challenges and defines just what we will be considering as text and image in the rest of the book. Unit 2 then shows more of the diversity of texts, images and text-image combinations that we will need to consider as well as working through some early accounts proposed for explaining how text and image work together. This then establishes some general goals for text-image research and description and sets the scene for the analyses and frameworks to be covered in subsequent parts of the book.

Module I Starting Points

1 Text-image relations in perspective

DOI: 10.4324/9781315773971-1

In many respects, and in many contexts, combining text and images is seen as the most straightforward, the most natural thing in the world. People have been putting textual information and images together for thousands of years – for just about as long as there have been textual information and images to combine, in fact. For all of this ‘naturalness’, it is equally common to hear of the ‘text-image divide’, of how text and image go about their business in very different ways. This is sometimes seen as a beneficial complementarity, at other times as a more competitive struggle for dominance. With such differences of opinion, it is perhaps not so surprising how much uncertainty and debate the practice of ‘combining’ text and image causes. The issue of ‘text-image’ relations frequently recurs in both academic and practical discussions. And, currently, it finds itself again at the peak of a wave of interest.

Many of the reasons giving rise to this continuing attention lie in the nature of the ‘objects’ being related: ‘texts’ and ‘images’ are generally addressed by very different groups of disciplines and those disciplines often take very different views both on just what they are describing and how descriptions should be pursued. But in order to relate things, there has to be some ground for comparison. And this can easily come to mean seeing texts more like images or images more like texts just so that comparison can get started. This can be a source of considerable friction: when those working on language are seen as telling those who work on images how their object of enquiry is to be investigated, or those working on images make statements about language and its ability or inability to offer useful views on images, the potential for conflict is considerable.

The purpose of this book is to take a fresh look at the questions and challenges involved in understanding how combinations of texts and images work. The book is divided into several modules, each leading us into different aspects of the text-image question. In this first module, we take our first look at the two domains that we will be concerned with – that is, text and images themselves. The main focus will be on how approaches to these areas construct their respective subject matters. We will bring together several disciplines, methods, frameworks and application areas in order to show text-image relations from several perspectives. The fundamental idea is that it is only with a broad approach of this kind that we can gather sufficient experience for the next round of text-image relation research.

MULTIMODALITY AND ‘MEANING MULTIPLICATION’

Text and images are sometimes referred to as different ‘modes’ of communication. We will refine this idea further as we proceed, particularly for the case of ‘images’, but this simple description already allows us to situate our task in this book against a particular background of research. We will be relating distinct modalities of information presentation. Text-image relations and their study consequently fall within the general area of multimodality – the investigation of diverse modes of expression and their combinations.

Combinations of this kind are sometimes considered in terms of meaning multiplication, a metaphor promoted by the socio-functional semiotician Jay Lemke (Lemke 1998). The idea is that, under the right conditions, the value of a combination of different modes of meaning can be worth more than the information (whatever that might be) that we get from the modes when used alone. In other words, text ‘multiplied by’ images is more than text simply occurring with or alongside images. In order to explore what kind of results can come out of such a multiplication, we need to have a better idea of just what is being multiplied. Somehow the meanings of one and the meanings of the other resonate so as to produce more than the sum of the parts. But in order for meanings to ‘build on’ each other in this way, there must be some contact, some internal properties that allow multiplication to take place. This raises some interesting theoretical challenges.

To suggest what is involved here, consider as an example apples and bananas. It is easy to add them – we do this whenever we buy them at the supermarket and put them in a shopping bag. The result is simple: we have an apple and a banana. What then could it mean to multiply them, to have more than just a collection of individual pieces of fruit (cf. Figure 1.1)?

There are a number of possibilities, some less far-fetched than others. Perhaps we use them as ingredients in a fruit salad and, somehow, the way their individual tastes work together achieves a certain new quality, different from the fruits when eaten alone – great chefs are probably exploring this new frontier all the time. Alternatively, perhaps we achieve a new hybrid fruit exhibiting interesting features of both by some process of genetic engineering. In each case, contact has been made at some deeper level between the apple and the banana (e.g., in the molecules responsible for their taste, or in their respective genetic codes) in order to create something new. The results in these hypothetical cases are not then simply apples and bananas mashed together, but new structural organisations recombining and recontextualising basic properties of the original fruits.

Figure 1.1 Multiplying meanings?

This is exactly what we will need to look for when we attempt to multiply text and images – otherwise we just have text and image in close proximity, without interaction and, therefore, without meaning multiplication. It is certainly possible to have such weak connections of text and image, but these alone would explain neither the long history of using text and image together nor the explosion in such practices that we are experiencing today. After all, if images and texts could do just as well by themselves, they would not need to be used together as often as they evidently are. Getting clear about this from the outset is very important for our current endeavour. Although it is a commonplace to suggest that text and image when used together somehow combine, ‘blend’, or ‘synthesise’ to give us new possibilities, we need to know how and in what ways this works (or does not work).

To explore this question well, however, we certainly need to avoid assuming that the two, images and texts, are more similar than they actually are – this would be equivalent to assuming that apples are in fact bananas (or vice versa) and so there is no problem combining them. Note that when we do this, we do not get anything new when we combine them – just more mashed bananas. Similarly, if text and images were more or less the same, then combining them also would not lead to anything substantially new. The most fundamental issue of all, therefore, when addressing text-image relations, is to find points of comparison within text and within image that are well motivated and productive rather than reductive and distorting so that one comes to look more or less like the other.

We must also take care not to go too far in the opposite direction. If text and images were ‘completely’ different, totally incommensurate, then combining them would not produce anything sensible either. Too many approaches fall foul of these extreme positions. They either claim text and image to have nothing in common – so researchers on images and researchers on texts should not even talk to each other – or assume images and texts are too similar, so there is nothing to learn from each other anyway!

Both strategies are disastrous because they give rise to walls between disciplines and approaches that are anything but helpful. The result is a kind of ‘disciplinary myopia’ where research questions and solutions develop (or not) in splendid isolation despite, in many cases, the existence of rather parallel concerns in other areas. As we proceed, therefore, we will practise being both more differentiating and more open so as to bring out just how texts and images may be similar (so that we can multiply them together) and how they are different (so that multiplication gives us more than addition). To do this, we will compare and contrast a broad range of approaches across disciplines and subdisciplines as we go.

GETTING STARTED: VIVE LA DIFFÉRENCE

The following quotation from the French linguist Émile Benveniste seems in many respects obviously correct and reiterates a position made by many researchers and theoreticians from diverse disciplines over the years:

Semiotic systems are not ‘synonymous’; we are not able to say ‘the same thing’ with spoken words that we can with music, as they are systems with different bases. In other words, two semiotic systems of different types cannot be mutually interchangeable.

(Benveniste 1986: 235)

Different ways of making meanings – which is what we can take ‘semiotic systems’ to be – indeed appear very different. But this raises our multiplication puzzle: if they are so different, how can they so readily combine?

It may appear obvious how to put some things together, and sometimes it might genuinely be obvious, but this cannot always be taken for granted. Will it always work? Are there situations when we need to be more cautious about putting different ways of making meaning together if we want to communicate successfully? And what kinds of theoretical tools and perspectives do we need for being able both to ask and answer these kinds of questions? These are the starting points we need to address before getting underway.

Issues

To refine these concerns and become more concrete, take a look at the map shown in Figure 1.2. We are generally familiar with this kind of representation, and this goes a long way towards helping us read it without too many problems. Although it is rather different from maps that we usually see these days, it is still obviously a map. This means that we can be fairly sure that the visually depicted areas have something to do with land and, between them, what we can see is probably the ocean.

We assume this for a number of reasons. On the one hand, we can assume this because of the rather more ‘wave’-like drawing in the ocean, together with something that looks like a whirlpool, some ships, and a sea creature; on the other hand, the white areas might be more or less familiar shapes of things we know as countries as well as sporting a few trees and buildings scattered around – and they also have some text on them, looking perhaps like names of countries, towns, and so on. Thus, because we know the kinds of things that the sea has (or has been thought to have), the kinds of things that we find on land, and are familiar with the practice of putting these onto flat pieces of paper, linen, canvas, and so on, we can ‘read’ what appears without apparent effort – that is, without apparently knowing much at all.

That this effortlessness is an illusion, however, is quickly revealed whenever we encounter representations with which we are not familiar. Then ‘just seeing’ is replaced by a more conscious attempt to work out what is going on. For example, if we were not familiar with old maps of this kind and their tendency to combine geographical detail with pictorial depiction, then we might well wonder what strange states of affairs are being reported – huge whirlpools due to global warming, perhaps? Far more likely for us is that other features of the map have already triggered the appropriate assignment of this representation to the category of ‘old maps’, since this is just not how maps are drawn these days. For example, the boats and (we can probably assume) the sea creature are evidently not intended to be read as being drawn to the same scale as the items shown on land. And neither of these is shown on the same scale as the land and sea themselves.

Figure 1.2 A map of what is to come?

In fact, more than this, all these distinct representations are not even built within the same ways of making meaning – they are pulling in quite different conventionalised directions for using visual depiction and combining them in the service of whatever it is that the map is intended to communicate. They are, to use the more technical term, drawn from rather different (but also related) semiotic systems. The overall result is coherent and communicates various messages, but it is put together from several distinct meaning-making systems. And here the more interesting questions start – just what are the various ways of making meanings employed here?

A rather naïve and commonsensical glance at the map might lead us to offer as a first guess: ‘words’ and ‘images’. But we have already mentioned that the depiction of the countries and their coastlines follows rather different rules from those of the ships and the dragons depicted in mortal combat in the ocean. Moreover, the words depicted are also clearly visual in that we can see them – the image pattern making up the shape ‘HELGALA’ is placed in the picture just as the wave tops are – and there are differences among these words that are not attributable to their linguistic properties as words at all. They vary in size, for example.

This means that even ‘within’ the visual information on offer, the particular forms dist...

Table of contents

• List of figures
• List of tables
• Acknowledgements
• Part I Relating Text and Image
• Part II Visual Contexts
• Part III Frameworks
• References
• Index