Sound and Action in Music Performance

Peter Q. Pfordresher

• Publisher: Academic Press
• Year: 2019
• ISBN:
  9780128094884
• Topic: Psychologie
• Subtopic: Angewandte Psychologie

Sound and Action in Music Performance addresses how auditory feedback influences the planning and execution of our movements. Focusing specifically on auditory feedback in music, including instrumental and vocal production, the book also gives substantial coverage to its role in speech. Both of these behaviors are the primary means by which people communicate their thoughts and feelings through the auditory modality, with auditory feedback being critical in each case. The book proposes that the role of auditory feedback emerges from the broader theme of coordination as our brain coordinates planned actions with concurrent perceptual events, including auditory feedback and other intrusive sounds.

Critically reviewing the existing literature and proposing hypotheses for future research, this book tackles a topic that has intrigued researchers for decades.

• Covers the role of feedback in event sequencing
• Details how motor systems influence the use of auditory feedback
• Tackles neural mechanisms for feedback processing
• Characterizes hierarchical representations and synchronization
• Addresses perception/action associations and the role of internal models of production
• Discusses how learning influences the use of auditory feedback
• Considers the role of feedback in music and speech production deficits

Chapter 1

Setting the Stage

Abstract

This chapter offers a general introduction to the book. It starts with an argument for why understanding the role of sound (auditory feedback) during music performance is interesting from a scientific and practical perspective. The chapter then reviews important concepts relating to the cognitive and neural bases of music performance and music perception, along with a description of critical concepts and paradigms used in the book. Most important, this chapter reviews theories of perception and action that have been used to explain how perception and action intersect in music performance, along with an initial description of a new theory of Hierarchical Perception/Action Coordination, which is developed across several chapters in this book (Chapters 1, 3–5, 8).

Keywords

Music performance; auditory feedback; planning; perception and action; closed- versus open-loop accounts of performance; EXPLAN; node structure theory

When the Beatles played concerts to crowds of screaming teenagers, they could barely hear themselves, and yet they sung in three-part harmony with astounding accuracy. On the other hand, speaking into a telephone with a delay in the sound can be tremendously disruptive. About 10 years ago, my university installed fancy “IP-drop” phones in all the offices. Although I enjoyed the sleek new design, my new phone had and still has a problematic quirk. Occasionally, while I’m talking, I hear a copy of my own voice that occurs a fraction of a second after I speak (this can occur when the microphone on the recipient’s phone picks up the output from your phone). The effect of this delay is maddening. When it happens, I become much more selfconscious of my voice. It becomes hard to pace my speech appropriately and find myself occasionally stuttering and drawling out my vowels, features of speech referred to as disfluencies. I find it impossible to ignore this rebroadcast of myself. Usually speaking at a slower rate helps a bit, but I almost always have to apologize to my interlocutor, who is unaware of the glitch. A roughly analogous effect used to occur for international calls, as there used to be a considerable delay in the time at which your phone would receive the signal from your interlocutor. In a twist of irony, I have used exactly this sort of manipulation on research participants for the past 20 years.

These examples concern the altered auditory feedback effect, a paradigm that will dominate the research discussed in this book. The alteration described above involves an alteration of timing—a delay in the onset of speech feedback relative to when speech is produced. However, alterations can be made to any feature of sound. Using modern digital technology, my voice could be altered to sound like another person’s, could be shifted in pitch, or could be made to sound like different words. Although the example I just gave has to do with speech, similar effects are found in music performance, which is the main focus of this book.

The effects of altered feedback can be entertaining or frustrating, but they also shed light on an important question: What role does perception have in performance? When you perform music, you have to plan, execute, and control a complex sequence of actions that can involve a sequence of finger movements (e.g., piano), or changing states of muscular tension (e.g., singing). The demands on the motor system, in the brain, muscles, and joints, are considerable. However, actions do not really constitute the goal of a musical performance. You perform music to create a pattern of sound that your listeners can perceive and enjoy. Thus, action planning in music performance may result from planning an intended sound pattern.

This book will address how the sound of one’s own performance (i.e., auditory feedback) influences the continued planning and the execution of music. Although my discussion will be dominated by research on the effects of altered auditory feedback, I will consider other paradigms that have to do with the role of sound in music performance. For instance, in Chapter 9 I discuss research on how performers use auditory information from each other to synchronize their performances. I will also consider the process by which people translate a melody into a sequence of actions when they play or sing “by ear,” a form of imitation. Although my main focus is on music, I will also consider the role of auditory feedback in speech (Chapter 7). Speaking and music making are the primary way people communicate their thoughts and feelings through the auditory modality, and thus auditory feedback is critical in each case.

For the most part, my discussions of music performance will concern the piano, with somewhat less focus on vocal performance, and unfortunately, very little on other kinds of instrumental performance. This limitation reflects the scope of music cognition research on performance. Piano outweighs other instruments by far in its popularity, and electronic pianos allow researchers to measure all relevant dimensions of performance with high efficiency and precision in the lab. Furthermore, the flexible mapping of actions to sound that is available on the piano make it an interesting test case for sensorimotor learning (as discussed further in Chapter 6). Although singing includes an even larger portion of the population (potentially everybody), the difficulty of measuring vocal production for some time inhibited its use in empirical studies. Another limitation of studying the voice, as I will discuss in Chapter 7, is the difficulty of implementing certain alterations to auditory feedback. Nevertheless, an increasing amount of research on singing has emerged, which I will discuss, which is fascinating in part due to the presence of large individual differences in pitch accuracy of singers (Berkowska & Dalla Bella, 2009, 2013; Dalla Bella, Giguère, & Peretz, 2007; Hutchins & Peretz, 2012a, 2012b; Pfordresher & Brown, 2007; Pfordresher & Larrouy-Maestri, 2015).

In this introduction, I lay out the basic territory that this book explores. I start by discussing the production of action sequences, focusing on music performance. I then discuss what we mean by perceptual “feedback” and consider some nuances associated with this term. Finally, I provide an overview of some of the leading theoretical and empirical treatments of perceptual feedback.

Intentions, Planning, and Execution in Music Performance

A great deal of psychological research considers discrete actions (e.g., a key press) or actions oriented toward a single goal (e.g., grasping a dowel). Such behaviors are highly useful given the amount of experimental control they allow, and the easy way that one can separate planning from execution. However, most of the time we plan and execute behaviors in a sequence. Even when grasping an object (e.g., a kettle) we are often doing so when preparing a subsequent goal (pouring into a coffee filter). As such, it is important for research on performance to consider the constraints on sequential actions.

Music is a quintessentially sequential behavior. A single note or chord means almost nothing outside of a context. An “F” may be a stable ending pitch in a happy tune (F-major), or it may cause a melody to sound sorrowful (D-minor), or may sound out of place (B-major). The sequence defines the meaning. A similar case can be made for speech. Single words can convey meaning (“Fire!”…“Yum!”). However, as in music the meaning of speech usually comes from sequential context (“The candidate lacks fire in the belly,” “I’m on fire with love for you,” or “Help, help…FIRE!”). Of course, there are many ways in which music performance differs from speech. I consider the similarities and differences between these two forms of expression in Chapter 7.

A particularly important component of music performance is the formation of an action plan: a series of motor commands based on an intended outcome. Let us now consider the link from intention to plan to execution. In so doing I will build the framework for a theory of perception and action in music that will be developed throughout this book: the Hierarchical Perception/Action Coordination (H-PAC) model.

First, what is a musical intention? Intentions are fundamental to psychology, and in many areas of behavior the notion of an intention is incredibly complex. What has a child intended when she throws a valuable china bowl on the floor? We have a somewhat simpler situation in music, though not without its own intricacies. For now I propose that a performer’s intention is an auditory image of a correct performance. In other words, musicians intend to create sound sequences, they do not directly intend to produce movements. Of course, there are some situations in which a performer may not have formed an auditory representation of the piece they want to play. This can happen if someone is reading a piece for the first time from notation, and the individual lacks the kind of reading skills that lead to associations of notation with auditory imagery (called notational audiation, Gordon, 1975). In such cases I would argue that the performer has not truly formed an intention for the piece of music, but instead is following a series of scripted commands. In other words, the notation in these cases stands as a kind of proxy for the intention. In Fig. 1.1, an intention is represented as notation for part of an intended melody (here, “Frère Jacques”) in a “thought bubble,” to convey the fact that the notation is being used to represent an imagined auditory pattern. In this book, when I refer to the planned melody in a performance, what I refer to is a produced melody that realizes this intention.

Figure 1.1 The formation of an action plan in music performance.

An important consideration regarding the role of imagery is that auditory images, like the music we perceive, are not accessible all at once. Auditory images are scanned in real-time, similar to the way visual images may be scanned (Halpern, 1988). Thus intentions may involve fully formed auditory images stored in long-term memory that are scanned through the next step in the process of performance: the formation and updating of an action plan.

Action plans act as an intermediate step between intentions and action, and also an intermediary between perception and action. First, action plans involve incremental retrieval from the intended sequence. During incremental retrieval an action plan is continuously updated while execution occurs. Incrementality is important for the production of any complex sequence, including speech and music (Wheeldon, Meyer, & Smith, 2002). Consider reading out loud from a book. The book provides a concrete abstract representation of your intention, which will be spoken prose that your audience can hear. Although you will execute your speech one word at a time, if you are a good reader your eyes will not fixate on the present word but instead will scan several words ahead. This allows you to form an action plan based on future words while you are executing the current word. This allows a reader to avoid common pitfalls like inappropriate pausing at the end of a line, failing to parse sentences appropriately, etc. A good reader plans her reading incrementally. In piano performances, the effects of incrementality can be seen in finger movements, which are initiated some time before the execution of the key press (Dalla Bella & Palmer, 2011; Engel, Flanders, & Soechting, 1997; Loehr & Palmer, 2007).

How does incremental retrieval work? Two key mechanisms have emerged in the literature: chunking and graded accessibility. Chunking is a well-known mechanism for coding information, proposed by George Miller as a way of enumerating the number of information channels available for short-term memory (Miller, 1956). Chunking, or at least an analogous process of segmentation into smaller units, also plays an important role in action planning for music performance. Patterns of timing variability (Van Vugt, Jabusch, & Altenmüller, 2012), retrieval failures (Palmer & van de Sande, 1995), and practice strategies (Williamon & Valentine, 2002) all suggest that musicians organize music into segments based on the hierarchical structure of music, most likely based on phrasing (cf. Lerdahl & Jackendoff, 1983). For instance, when pianists make serial ordering errors in music, the misplacement of musical events (notes or chords) tends to occur within a musical phrase, only rarely crossing the boundary into an adjacent phrase (Palmer & van de Sande, 1995).¹ In Fig. 1.1, the process of chunking is represented by placing brackets around a plausible “chunk” within the music being represented.

One difficulty in applying traditional chunking theories to music is that chunks in standard memory models involve all-or-none retrieval. Remember that the idea of chunking was originally introduced to explain why we have no more problem retrieving a list of words with many letters (apartment, refrigerator, automobile), than one with words having few letters (car, house, stove). This is because information is coded as discrete chunks, so what you recall is a word and not letters within a word. Retrieval of a musical phrase is not directly comparable to this scenario. Thus, for music there is graded accessibility. One model of graded accessibility, which plays a central role in this book, is the range model of planning (Palmer & Pfordresher, 2003; Pfordresher, Palme...

Table of contents

• Cover image
• Title page
• Table of Contents
• Copyright
• Preface
• Chapter 1. Setting the Stage
• Chapter 2. Do We Need Auditory Feedback? If Not, Why Not?
• Chapter 3. Binding Perception and Action in Time
• Chapter 4. Binding Planned Actions to Their Consequences
• Chapter 5. Timing Versus Sequencing in Music
• Chapter 6. Effects of Musical Training
• Chapter 7. Use of Sound in Speech Versus Music
• Chapter 8. Auditory Feedback and Higher Cognitive Functions
• Chapter 9. Coordinating With Others
• References
• Epilogue: What We Have Learned and Where to Go
• Author Index
• Subject Index