Body, Sound and Space in Music and Beyond: Multimodal Explorations

Clemens Wöllner, Clemens Wöllner

• Publisher: Routledge
• Year: 2017
• ISBN:
  9781317173465
• Edition: 1
• Topic: Media & Performing Arts
• Subtopic: Music

Body and space refer to vital and interrelated dimensions in the experience of sounds and music. Sounds have an overwhelming impact on feelings of bodily presence and inform us about the space we experience. Even in situations where visual information is artificial or blurred, such as in virtual environments or certain genres of film and computer games, sounds may shape our perceptions and lead to surprising new experiences. This book discusses recent developments in a range of interdisciplinary fields, taking into account the rapidly changing ways of experiencing sounds and music, the consequences for how we engage with sonic events in daily life and the technological advancements that offer insights into state-of-the-art methods and future perspectives. Topics range from the pleasures of being locked into the beat of the music, perception–action coupling and bodily resonance, and affordances of musical instruments, to neural processing and cross-modal experiences of space and pitch. Applications of these findings are discussed for movement sonification, room acoustics, networked performance, and for the spatial coordination of movements in dance, computer gaming and interactive artistic installations.

1 Introduction

Structured sounds in bodily and spatial dimensions

Clemens Wöllner

The key terms in the title of this volume address far-reaching concepts that have been investigated in music psychology and musicology for a long time. Given the wealth of research, each term would perhaps merit a book on its own. Body and space refer indeed to important and interrelated dimensions in the experience of sounds and music. Most notably, in most musical cultures and genres it is originally the human body in interaction with physical objects that produces musically meaningful sounds. Besides a richness of instruments as external objects, the human voice has long been considered to be the earliest instrument and the origin of music-making (Rousseau, 1786; Mithen, 2006). Hearing someone’s voice or listening to a musical instrument playing may resonate with the listener’s motor system (Sevdalis and Keller, 2014), thus enabling embodied experiences of musical sounds that some listeners even intensify by moving their body along with the music, ranging from armchair conducting to fine-tuned or frantic dancing in space with the whole body.

Recent advances in methodologies and technical devices permit researches to scrutinise these central experiential dimensions in a detail that was unthinkable a few decades ago. Real-time sound analysis and modelling, three-dimensional motion capture of body movements in high temporal and spatial resolution and immersive explorations of virtual performance spaces – to name but a few of these developments – lead to an unprecedented richness of data. The more information is gathered in diverse and specialist fields, the more there is a need for approaches that compare and coalesce the findings, and relate them to overarching theories that may reconcile apparently contrasting research perspectives. For example, physical descriptions of space and sounds may fundamentally differ from their perceptual counterparts and from meaning formation. Sound perception depends on individual psychological representations and not least basic physiological differences between the hearing systems of listeners. Moreover, there are to date no all-encompassing and meaningful approaches to capture apparently measurable facets of sound such as timbre in acoustic dimensions (see Reuter and Siddiq, this volume). In a similar vein, there are concepts of metaphorical space (e.g., the wideness of sounds), cross-modal perceptions (e.g., high and low for auditory pitches) and multimodal perceptions (e.g., visual, haptic, proprioceptive, auditory) of spatial dimensions based on Gestalt laws, all of which may differ from outer space and three-dimensional Euclidian measures (Eitan, 2013). In an early approach, Ernst Kurth (1931, p. 116) considered the experience of space as a ‘central function in listening’ to music and a valid psychological reality, such that even the demarcation of sound in distinct pitches should already evoke spatial images.

Experiences of sound and music, as a consequence, are genuinely multimodal by evoking vivid connections to bodily and spatial representations. Sound perceptions are not limited to afferent information via unimodal sensory receptors, but are rather experienced in the full associative power that the same sounds represent in real-life situations (cf. Clarke, 2005), resulting in the need to capture both stimulus features and representational parameters in the individual. Regarding the latter, some of the multimodal perceptions are based on higher-order cognitive associations grounded in memory processes, while other perceptions function on a more automatic level involving bodily representations that may not always be experienced in full awareness (cf. Cross, 2010). Therefore, conscious perceptions of the experiential dimensions alone may not lead to the full picture, and indirect measures have been included in research.

Investigating body, sound and space opens the field for approaches to one of music’s fundamental features – its motional character with relevance for listeners, composers, theorists and performers. As David Epstein (1995, p. 101) put it, ‘motion subsumes, integrates, and provides the broadest context for all other musical elements’. Characterisations of music as ‘moving sonic forms’ (Hanslick, 1854/1891, p. 74) or ‘features of musical sounds in relation to their gestural affordance … as trajectory shapes in time and space’ (Godøy, 2010, p. 113) point to perceptions of motion qualities within music that are mediated by the human body (cf. Kurth, 1931; Truslit, 1938; Leman and Maes, 2014). Embodied approaches (e.g., Gallagher, 2005) lay stress on these mediation processes that are seen as interactions between the body and the environment. An example is the relationship between bodily reactions and the experience of being moved by music, such that the musical sounds resemble the time course of emotional responses (cf. Langer, 1942). Pratt (1931/1968, p. 203) stated that auditory characters are not emotions themselves, ‘they merely sound the way moods feel’. The perceived motion of auditory characters in music may thus correspond with bodily movements, and emotional experiences of music often involve dynamic components of these movements (Epstein, 1993; Davies, 1994; see Eitan, this volume).

A further important aspect – time – could perhaps have been added to the title of the book. Music as a temporal art and, more broadly, sounds as ephemeral sonic events are already intrinsically defined by timing dimensions. Indeed, many contributions to this volume deal with the time course of sounds. The word ‘beyond’ in the title is chosen to include various types of music and sound experiences that are not often dealt with in musicological and music-psychological writings, such as sound installations (e.g., Licht, 2007), films or video games, for which the experience of space and body are equally important. Taking the perspectives of the creators or producers, sounds that are structured deliberately by a composer, improvising musician or sound designer are typically regarded as music. In a similar way, even sounds that are not intentionally composed can be experienced as intentional music if the listeners’ memory system restructures their perception (such as, for example, in Diana Deutsch’s well-known ‘speech-to-song illusion’; see Deutsch, Henthorn and Lapidis, 2011).

The book may not ultimately clarify and stipulate the relationship between these key concepts, yet it is important to shed light on them every now and then, such as Kurth (1931) and other music scholars did, employing the methods available at their time. This volume provides a platform for the discussion of new developments, taking into account the rapidly changing ways of listening and experiencing sounds, recent theoretical and empirical approaches, consequences for how people engage with sound events in daily life as well as technological advancements that offer insights into state-of-the-art methods and future perspectives.

Overview of Body, Sound and Space in Music and Beyond

The chapters of this volume discuss theories and recent research from a wide range of disciplines, including music psychology, composing, musicology, computer science, music theory, sound arts, acoustics and neuropsychology. Since multimodal experiences of sounds and music are not limited to the concert hall or silent listening at home, but have an impact on a great variety of activities and experiences in our daily lives, multidisciplinary approaches are necessary for capturing both the fundamental psychological processes involved and the specific experiential qualities for the individual engaging with music and sounds. Topics range from theories of pleasure in bodily imitation, bodily resonance in relation to accounts of perception–action coupling, investigations of spatial dimensions of sound in virtual environments or installations, through to neural mechanisms of space and sound processing or topics addressing the embodied perception of sonified movements and many more. By bringing together these perspectives, a great number of links between themes becomes apparent. For example, bodily experiences of sounds are often described using metaphors that reflect spatial dimensions. Sounds may have an overwhelming impact on feelings of presence and inform us about the space we experience, even in situations where visual information is artificial or blurred such as in virtual worlds, certain film genres or computer games.

Part I: Bodily movements, gestures and sonification

The first part of the book highlights the significance of music-inherent gestures and bodily movements for experiencing sounds and music. All chapters in this section also refer to spatial dimensions and characteristics of sound. The main perspective of the contributions is based on theories of embodied cognition and perception–action coupling.

Marc Leman, Jeska Buhmann and Edith Van Dyck investigate in Chapter 2 ‘The empowering effects of being locked into the beat of the music’. Based on studies carried out in their own lab and elsewhere, they describe two key mechanisms for synchronisation between beat patterns in music and human movements: predictive processing and reward processing. The first mechanism is based on dynamic interactions between the human body and auditory sound events, while the second mechanism draws on perceptions of agency leading to pleasurable feelings of reward. Both processes give rise to the sensation of empowerment when synchronising with a beat. The argument is supported by examples from a system called D-Jogger, where the phase and period of the music can be adjusted according to the human rhythm of the footsteps. This system allows various degrees of synchronisation and spontaneous movement adjustments, depending on the quality of sounds to synchronise with as well as individual characteristics of the human body.

A different look at bodily processes is taken by Alexander Refsum Jensenius in Chapter 3, who explores micromotion and microsound. In a series of experiments on ‘standstill’ behaviour, individuals were asked not to move at all, either while listening to music or being in silence. This approach on the boundaries between science and artistic experiences explores the tiny micromovements over time that are hardly visible, yet may tell researchers a lot about voluntary and unconscious interactions with music, sound and silence. With reminiscences to concepts of Cage, insights into bodily behaviour were gained by recording the standstill behaviour of various people including a dancer and musicians with a motion capture system. The performances reveal consistent patterns across sessions, indicating an extraordinary ability of the body to adapt to different situations. In contrast, subjective experiences of the tiny movements varied according to the sound and music. Besides the novel approach to micromotion, this chapter also explores bodily behaviour in joint situations.

Zohar Eitan approaches a key dimension of musical sound and motion in Chapter 4: pitch height and multimodal experiences of changing pitches. The cross-modal nature of many descriptions of pitch becomes evident when considering the multitude of words used to describe pitch changes, showing some similarities to descriptions of bodily states and spatial dimensions. Among them, Eitan highlights cross-modal perceptions of motion, touch and vision, and focuses on these interrelations by investigating the closeness of multimodal mappings in terms of valence and affect or static versus dynamic associations. In particular, experiences related to dynamic changes provide insight into perceptions that clearly attribute spatial and even bodily qualities to pitches. Investigating these dimensions of pitch and musical motion sheds light on fundamental questions in music cognition and theory.

In Chapter 5, Clemens Wöllner and Jesper Hohagen discuss evidence for correspondences between inner musical motion and outward bodily movement responses. Following an overview of research on the gestural quality of music, including spatiotemporal perceptions, kinaesthetic sensations and imagery, they investigated ideas of Alexander Truslit (1938), who claimed that there are dynamic changes in musical interpretations that can be perceived by listeners in an ‘armchair conducting’ style. In a series of studies, participants moved their arm along with original musical examples of Truslit, while their movements were recorded with a motion capture system and subsequently sonified using a parameter-mapping approach. High individual movement consistency was found across different listening conditions, comparable to Jensenius’ findings (this volume). Selected sound characteristics were related to participants’ spatial position data, supporting the interpretation that sound can be related to musical experiences as expressed in bodily motion.

The final chapter in this section deals with aesthetic qualities of sonification and explores human–machine interactions. Paul Vickers, Bennett Hogg and David Worrall explain the ideas, technologies and challenges of sonifying data and in particular human bodily motion. Comparable to types of visualisations that highlight otherwise unseen aspects, sonifications are used in many contexts, including arts, sports or medicine. Quite often, nevertheless, sine-wave sounds or other rather artificial sounds have been used that are not always aesthetically pleasant or effective. Investigating the aesthetics of sound qualities in sonifications is not only a highly needed endeavour, it may also provide insights into perceptions of sounds more generally, including their multimodal associations. The authors concentrate on the meaning and intentional qualities of sounds as a result or a by-product of actions, in contrast to more specific parameter-mapping approaches in sonification. In the latter, individuals may have some control over the sound quality they produce or experience with their bodies.

Part II: Sound design, instrumental affordances and embodied spatial perception

The second part of the book addresses multimodal perceptions of sound in relation to spatial dimensions – from the spatial design of musical instruments and its consequences for sound production up to metaphors of sound and space. As stated above, structured sounds may be perceived as music, yet the specific quality of sounds and their relation to meaning formation and bodily experiences have largely been neglected in writings on music.

Spatial and bodily dimensions in musicians’ performative actions are discussed by W. Luke Windsor in Chapter 7. Based on theories of ecological perception by Gibson (1979) as well as perception–action coupling, music performance is seen as closely related to the human body in the fields of acousmatic compositions or free improvisations, with conventional instruments and in vocal performance. A key term in this regard is the concept of ‘affordances’, which relate to the possibilities inherent in the instrument which may influence the musicians’ actions. Two examples highlight the bodily affordances of instrument design (the Böhm flute) and bodily as well as spatial characteristics of a group performance (Barbershop chorus). Windsor argues that the performative actions of musicians are formed by cultural norms and the musicians’ empirical investigations, seeking contact between the body, the instruments and the performance environment. Interpretations of the instrumental affordances and bodily processes may inform audience responses in similar ways as socially constrained norms and performance spaces.

In Chapter 8, Kathrin Fahlenbrach approaches the role of sound design and music in the creation of spatial experiences in film. Sounds have a crucial role not only for guiding the listeners’ attention and expectancies, but also for memory processes and emotional responses. Based on Chion’s (1994) concept of synchresis, combining temporal synchronisation and semantic synthesis, Fahlenbrach explores multimodal metaphors (see Eitan, this volume) that are crucial for the formation of meaning in film. In this way, the sound design of films establishes visual and acoustic Gestalts that shape perceptions of cross-modal correspondences and embodied meaning. In four case studies, spatial Gestalts are highlighted as metaphoric source domains, exemplifying both lower-level perceptual processes and the creation of more complex narrative and cultural meanings, with a specific focus on how sound may shape the perception of atmospheres and space.

Spatial dimensions of sound qualities are the starting point for Christoph Reuter and Saleh Siddiq, who describe the ‘The colourful life of timbre spaces’ in Chapter 9. Theoretical and empirical research on instrumental timbre generally failed to provide overarching categories for their description. Acoustical analyses alone do often not correspond with listeners’ aesthetic experiences of sound. Reuter and Siddiq summarise writings from the 18th century, followed by von Helmholtz’s (1877) system of the sound spectrum up to the most recent experimental approaches. ‘Timbre spaces’ are now a well-established paradigm to present the dimensional qualities of sound along three or more axes. Within these dimensional models, the closeness or distance of various sounds is depicted at certain points in space. Using computational methods derived from Music Information Retrieval, a meta-timbre space is suggested that overcomes some of the previous models’ shortcomings ...

Table of contents

• Cover
• Half Title
• Title Page
• Copyright Page
• Table of Contents
• List of figures
• List of tables
• Series editors’ preface
• Notes on contributors
• 1 Introduction: structured sounds in bodily and spatial dimensions
• PART I Bodily movements, gestures and sonification
• PART II Sound design, instrumental affordances and embodied spatial perception
• PART III Presence and immersion in networked and virtual spaces
• Index