Multimodality, Learning and Communication

A social semiotic frame

Jeff Bezemer, Gunther Kress

• Publisher: Routledge
• Year: 2015
• ISBN:
  9781317418429
• Edition: 1
• Topic: Languages & Linguistics
• Subtopic: Linguistics
• Index: Languages & Linguistics

This state-of-the-art account of research and theorizing brings together multimodality, learning and communication through detailed analyses of signmakers and their meaning-making in museums, hospitals, schools and the home environment.

By analyzing video recordings, photographs, screenshots and print materials, Jeff Bezemer and Gunther Kress go well beyond the comfortable domains of traditional sites of (social) semiotic and multimodal research. They steer away from spurious invention and naming of ever more new and exciting domains, focusing instead on fundamentals in assembling a set of tools for current tasks: namely, describing and analyzing learning and communication in the contemporary world as one integrated field. The theory outlined in the book is grounded in the findings of the authors' wide-ranging empirical investigations. Each chapter evaluates the work that is being done and has been done, challenging accepted wisdom and standing much of it on its head.

With extensive illustrations and many examples presented to show the reach and applicability of the theory, this book is essential reading for all those working in multimodality, semiotics, applied linguistics and related areas. Images from the book are also available to view online at www.routledge.com/9780415709620/

1 Recognition

10.4324/9781315687537-1

A multimodal perspective

As an introduction to the framework we develop in this book, consider Figures 1.1 and 1.2. Figure 1.1 shows some medical students standing around an operating table. The operation that is taking place is a laparoscopic procedure, which means that the surgeon uses a video camera to see inside the patient’s body. Figure 1.1 shows the screen to which the camera is connected, and it shows that everybody is looking at that screen. Figure 1.2 shows a snapshot of the view of the camera projected on the screen. It shows the operating surgeon pushing the liver to one side with an instrument, at the same time as he says ‘That’s the liver’.

Figure 1.1 Medical students (foreground) and surgeons (background) looking at a screen during an operation.

Figure 1.2 Snapshot from the screen as the operating surgeon says, ‘That's the liver’.

The example illustrates, first of all, how the people there use a range of semiotic resources available in this environment to communicate. In this case, the surgeon uses gesture to point at, and speech to name, an anatomical entity. He needs gesture as much as speech to draw the medical students’ attention to an organ in this patient, and to identify it as ‘the liver’. Without either mode of gesture or speech, this brief teaching episode would have been rather different. Naming alone would leave the students with the job of establishing exactly what, in this ‘mess’ of ‘stuff’ inside the patient’s abdomen, counts as ‘the liver’. Pointing alone would leave the students with the job of establishing what the object is that is marked out by the pointing – that is, what anatomical category it is an instance of. The students, for their part, do not speak at all; they engage with the object at issue through gaze, which is also how they communicate to the operating surgeon that they are attending to what he is highlighting. All are positioned so as to have sight of the screen, looking at the liver.

Second, the example illustrates how semiotic resources are used to teach. It highlights that teaching is an instance of multimodal communication. Here, the consultant uses a range of different communicative resources to design a multimodal learning environment. The surgeon’s pointing gesture and naming suggest that he has designed an environment for the students to engage with, in which the surgeon performs, and the students watch. This social/pedagogic relation is materialized in other modes, too, such as dress, for instance. The surgeon wears a gown, allowing him to enter the sterile zone and touch the patient, while the students do not; they only wear scrubs (i.e. non-sterile undergarments). The relation is also evident in room layout and in the positioning of participants: the surgeon is positioned at the centre of the operating theatre and stands centrally at the operating table, whereas the students are placed somewhat more peripherally to the operating table. In addition, there is a differential use of modes: the surgeon speaks and points, while the students remain silent and hold their hands behind their backs.

Third, the use of semiotic resources provides inroads into learning. For instance, the medical students’ (body) positioning and gesture suggest knowledge about where to stand and where and how to look and where not to stand and look. Through the mode of dress, they signify knowledge of how to wear scrubs and tie a ‘hat’; they signify knowledge of when to speak and to whom, when not to speak, how to speak, and so forth. If we were to trace these signs of learning over time, we would begin to see how, through a process of transformative engagement, the medical students gradually expand their resources for making meaning and acting in this environment.

In our account here, we break with the tradition of conflating learning with the presence of an explicit, institutionally designed and located curriculum. Rather, learning is evidenced in every sign produced, not by a pre-defined, selective subset of signs. While we recognize that some learners may not (yet) have learned what the (overt) curriculum prescribes, and concur that it is in their interest to assess ‘their learning’ against such a yardstick, we want to highlight the extent of the domain and the resourcefulness of the people there (and of people as learners in general) in relation to learning. To recognize this resourcefulness, we need lenses and methods that move well beyond contemporary metrics, such as a linguistic lens and psychometric tests, for instance. In this book, we hope to go some way towards sketching what such means of recognition and (alternative) forms of assessment and (e)valuation might be and look like.

Communication as learning, learning as communication: an encompassing framework

In this book, we pursue two ambitious aims. First, we develop a theory that shows how communication and learning are interlinked, mutually constituting and defining of each other in a closely integrated domain of meaning-making. Second, we show how such an account can provide an encompassing frame for describing different domains of social practice and semiotic work. In doing so, we illuminate learning and communication in distinctly new ways. In other words, we set out a theory that encompasses, marks out and accounts for a large field of social semiotic practice; at the same time, we introduce tools that the theory produces and that serve to provide descriptions, integrated analyses and insights for this field.

We think that the most significant effect of this approach may well be in relation to our first aim: to show that areas that at the moment are seen as distinct – in practical, professional and therefore often also in disciplinary terms – are in fact closely related and integrated. Activities that at the moment are dealt with under headings such as ‘communication’ and ‘learning’, and by different disciplinary approaches – sociology, psychology, anthropology, pedagogy, linguistics and semiotics, among others – can, at a more general level, be seen to form an integrated field of meaning(-making). And although we deal with the issue in only the briefest way in the book, we point towards closely related topics, such as socialization and identity. The benefit of this approach lies in showing the intrinsic relation and connectedness of fields that have hitherto been the subject of the attention of distinct disciplines with partially overlapping interests yet distinct framings. The larger frame provides possibilities for better, more satisfactory, integrated accounts. So, for instance, showing the fields of communication and learning as belonging in one theoretical frame has far-reaching beneficial effects for each.

The configuration into one large and integrated frame has repercussions for thinking and working in all of the domains of the social, wherever communication and learning in any of their manifestations are the issue. We expect that the explanatory power of such an approach will have effects on how ‘the world itself’ is seen and described, and will shape how appropriate disciplines can and will develop. What we are proposing is in large part a response to changes in the social (economic, technological) world: it takes the world as it has become and is now, and as present trends suggest that it will further develop. Connected closely with changes in that world comes an awareness that the disciplines, in their present form, are no longer capable of addressing the tasks that are there to be dealt with now. What, for instance, is the difference between learning and socialization, now that education has escaped the walls of existing, often venerable institutions? Or, a different question: how are we to think about the resources we have for communication, when the centrality of speech and writing is challenged everywhere, on countless occasions, in nearly all social domains? What does ‘learning’ mean, when formalizations of what is to be learned (curricula) are no longer confined to institutional frames and framings, and cannot be defined within the framings of now hugely diverse states and societies; when success in ‘learning what is to be learned’ is assessed outside the traditional frames and their metrics of power, outside the walls of institutions? What, now, is the place of design, when everyday, banal communicational tasks are invariably complex and unpredictable, and therefore subject to requirements of rhetoric and design?

To achieve our aims, we need, first, to respond theoretically to the social in which we now find ourselves; second, to describe the cultural/semiotic means and processes available for making meanings in contemporary social sites; and third, to describe the semiotic work done by those who act in the various domains, with the resources needed and available there. To do so, we look at a range of sites, including institutions (such as schools and hospitals) and other everyday sites and occasions of communication and learning. The range of our examples is in no way exhaustive but may be inclusive enough to indicate the outlines of the frame and of the task. We need – to repeat this once more – to expand our frame of attention significantly and begin the work of producing the requisite tools.

A generosity of recognition

The framework we develop is intended to have real effects on what and whose semiotic work is to be recognized, particularly work that at the moment is often disregarded, ‘invisible’ maybe – work that goes unnoticed, or is simply taken for granted. We are proposing a framework in which all means for making meaning become visible and recognizable – a framework that gives recognition to agency, and to identity, to ways of knowing and to learning of all kinds, everywhere. In doing so, we begin the task of making what is currently unnoticeable noticeable, what is inaudible audible and what is invisible visible. It is a task that will at the same time show much of what we know already, though in a markedly different light.

This generosity of recognition vastly expands the scope of what is given attention, with significant effects. If signs of learning are not recognized, they can be neither evaluated nor, therefore, valued. If (institutional) authority is blind to certain means for making meaning, then those who use these means are placed outside the domain of recognition. If (institutional) authority is regarded as paramount, then the principled transformative engagement of all learners and of most learning is ruled out of court.

Socially, these are questions of politics and power; semiotically and pedagogically, they are translated into valuations of different ways of seeing the world. If we wished (educational) institutions to continue to produce conformity and adherence to convention, then two things would need to be maintained. First, the canonically recognized means of making meaning would need to continue to be supported as canonical; other means would need to continue to be marginalized or kept invisible. Second, theories of learning and communication that privilege authority and its power would have to be defended. This might ensure that notions of ‘correctness’ and ‘error’ could prevail: ideologically limited conceptions of innovation and creativity would continue to serve the interests of power.

If, however, it is our aim to understand the constantly transformative and innovative character of human meaning-making, and therefore of learning, then all means used in making meaning will need to be recognized. All signs will need to be taken seriously, regardless of who made the sign, or in what mode; its valuation in the environment in which it was produced will need to be examined and understood. Instead of dismissing signs as ‘errors’ and sign-makers as ‘incompetent’, assessors will be required to investigate and establish the semiotic principles applied by sign-makers, to describe their ‘resourcefulness’ (Mavers 2007) and to explore and document their creativity in conjoining meanings with forms, in instances of the banal and everyday. Creativity and innovation will need to be seen as the ordinary, banal, constant processes and phenomena that they are. It will be in the illumination of the principles at work that their enormous richness and potential will be revealed.

We know, accept and understand that in many institutional settings, including institutions for learning, some signs are promoted and others discouraged or even banned; we do not wish to suggest that in these or in any other settings there should be an entire ‘free-for-all’. Ethical principles of semiotic work would need to exist and be understood and accepted in order to prevent or limit harm. Yet if the aim of some institutions is to design and foster environments in which sign-makers can expand their semiotic repertoires, then the fullest possible insight into the principles underpinning their sign-making is essential. The role of the interest of sign-makers, and the consequent exploration of the principles at work in transformation, interpretation and meaning-making, will need to be acknowledged and explored. A focus on the description and evaluation of interest and the elucidation of principles at work in learning will lead to profoundly different conceptions – and metrics – of (e)valuation than those of conformity to power, which have hitherto been present and dominant both in the educational mainstream and in a more broadly commonsense understanding. That is the case whatever the environments might be.

The theory we aim to develop will need to reconcile two larger-level principles, with seemingly contradictory tendencies: first, a theoretical framework of general semiotic principles which maintains, fosters and accounts for all culturally specific practices; and second, a means of mediating the presence of power in the face of common participation in social practices.

Describing the multimodal semiotic world: terms and categories

Throughout, one constant concern is – to use a term we take from the work of Basil Bernstein (1996) – to develop an apt language of description. We are putting forward a theoretical frame adequate for dealing with our main aims and the issues connected with them, as they appear in contemporary social practices, sites and events. For that, the categories and terms that we have inherited from existing disciplines are, by and large, no longer adequate. Some categories and terms may need to be jettisoned entirely. Others will need to be re-shaped, re-cast, and supplemented with categories and terms that are and will be newly made.

Taking the notion of multimodality seriously means finding, constructing and using terms that – at some level of generality – apply to and encompass the characteristics and potentials of all modes. At the same time, another set of terms is needed which names these shared – general – characteristics in ways specific to each mode. Moreover, the theory requires terms that adequately fit a variety of configurations of modes – that is, that fit modal ensembles. At the moment, we tend to ‘make do’ with selected instances only...

Table of contents

• Cover
• Halftitle Page
• Title Page
• Copyright Page
• Contents
• Figures
• Preface
• 1 Recognition
• 2 Sign-making
• 3 Transformative engagement
• 4 Shaping engagement
• 5 Assessment and judgement
• 6 Gains and losses
• 7 Applying the framework
• Appendix: list of research projects
• References
• Index