Intelligent Music Production

Brecht De Man, Ryan Stables, Joshua D. Reiss

• Publisher: Focal Press
• Year: 2019
• ISBN:
  9781351679022
• Edition: 1
• Topic: Technology & Engineering
• Subtopic: Acoustical Engineering
• Index: Technology & Engineering

Intelligent Music Production presents the state of the art in approaches, methodologies and systems from the emerging field of automation in music mixing and mastering. This book collects the relevant works in the domain of innovation in music production, and orders them in a way that outlines the way forward: first, covering our knowledge of the music production processes; then by reviewing the methodologies in classification, data collection and perceptual evaluation; and finally by presenting recent advances on introducing intelligence in audio effects, sound engineering processes and music production interfaces.

Intelligent Music Production is a comprehensive guide, providing an introductory read for beginners, as well as a crucial reference point for experienced researchers, producers, engineers and developers.

Part I

What Do We Already Know?

1

Introduction

“Shannon wants to feed not just data to a Brain, but cultural things! Hewants to play music to it!”

Alan Turing (father of modern computing) about Claude Shannon (father of information theory), during his 1943 visit to Bell Labs

In this chapter, we give an overview of the history and the state of the art in intelligent audio and music production, with a particular emphasis on the psychoacoustics of mixing multitrack audio, automatic mixing systems and intelligent interfaces.

1.1 Intelligent Music Production – An Emerging Field

Recent years have seen the emergence of intelligent systems aimed at algorithmic approaches to mixing multitrack audio with only minimal intervention by a sound engineer. They use techniques from knowledge engineering, psychoacoustics, perceptual evaluation and machine learning to automate many aspects of the music production process.

An increasing number of companies – from startups to major players in audio software development – have released new products featuring high-level control of audio features, automatic parameter setting, and full ‘black-box’ music production services. Many of the recent conferences, conventions and workshops by the Audio Engineering Society have featured dedicated sessions on topics like semantic music production or intelligent sound engineering. All of this leaves little doubt about the importance, in both academia and industry, of the wider field of analysis and the automation of music production processes. Existing approaches to problems in these key areas are underdeveloped, and our understanding of underlying systems is limited.

For progress towards intelligent systems in this domain, significant problems must be overcome that have not yet been tackled by the research community. Most state of the art audio signal processing techniques focus on single channel signals. Yet multichannel or multitrack signals are pervasive, and the interaction between channels plays a critical role in audio production quality. This issue has been addressed in the context of audio source separation research, but the challenge in source separation is generally dependent on how the sources were mixed, not on the respective content of each source. Multichannel signal processing techniques are well-established, but they are usually concerned with extracting information about sources from several received signals, and not necessarily about the facilitation or automation of tasks in the audio engineering pipeline, with the intention of developing high-quality audio content.

Thus a new approach is needed. Our goal in most music production tasks relates to the manipulation of the content, not recovering the content. Intelligent Music Production (IMP) has introduced the concept of multitrack signal processing, which is concerned with exploiting the relationships between signals in order to create new content satisfying some objective. Novel, multi-input multi-output audio signal processing methods are required, which can analyze the content of all sources to then improve the quality of capturing, altering and combining multitrack audio.

This field of research has led to the development of innovative new interfaces for production, allowing new paradigms of mixing to arise. However, it has also uncovered many gaps in our knowledge, such as a limited understanding of best practices in music production, and an inability of formal auditory models to predict the more intricate perceptual aspects of a mix. Machine learning has shown great potential for filling in these gaps, or offering alternative approaches. But such methods often have the requirement of being tailored towards problems and practical applications in the domain of audio production.

The following sections present an overview of recent developments in this area.

1.2 Scope

The music lifecycle, from creation to consumption, consists loosely of composition, performance, recording, mixing, mastering, distribution and playback. For the initial music creation stages of the workflow, generative music and algorithmic composition have shown the potential of autonomous music creation systems. Papadopoulos et al. [1] provide an excellent review of twentieth century approaches, and more recently there has been a surge in generative and automatic music composition for everything from elevator music to advertising. It is highly likely that this will become an increasing part of casual music consumption. The latter stages of the workflow, related to music consumption, have already been transformed. On the distribution side, musicians can share their own content at very little cost and effort, and intelligent recommendation systems can find preferred content and create bespoke playlists based on the user’s listening history, environment and mood.

1.2.1 Intelligent

By intelligent, we mean that these are expert systems that perceive, reason, learn and act intelligently. This implies that they must analyze the signals upon which they act, dynamically adapt to audio inputs and sound scenes, automatically configure parameter settings, and exploit best practices in sound engineering to modify the signals appropriately. They derive the processing parameters for recordings or live audio based on features extracted from the audio content, and based on objective and perceptual criteria. In parallel, intelligent audio production interfaces have arisen that guide the user, learn their preferences and present intuitive, perceptually relevant controls.

1.2.2 Music

Many of the concepts described herein might also be widely applicable in other audio production tasks. For instance, intelligent mixing technologies could have strong relevance to game audio, where a large number of sound sources need to be played simultaneously and manipulated interactively, and there is no human sound engineer in the games console. Similarly, they are relevant to film sound design, where Foley, dialog and music all need to be mixed, and low budget film and TV productions rarely have the resources to achieve this at a very high standard. However, to keep focus, we assume that the problems and applications are restricted to music.

1.2.3 Production

While there is overlap, we are not specifically referring to music creation – the composition and performance. Nor are we concerned with the distribution and consumption that happens after production. The middle stages of the music workflow – recording, mixing and mastering – are all about the production of the music. They are concerned about how the creative content should be captured, edited and enhanced before distribution.

1.3 Motivation and Justification

The democratization of music technology has allowed musicians to produce music on limited budgets, putting decent results within reach of anyone who has access to a laptop, a microphone, and the abundance of free software on the web [3,4]. Despite this, a skilled mix engineer is still needed in order to deliver professional-standard material [5].

Raw, recorded tracks almost always require a considerable amount of processing before being ready for distribution, such as balancing, panning, equalization, dynamic range compression, and artificial reverberation to name a few. Furthermore, an amateur musician or inexperienced recording engineer will often cause sonic problems while recording. As noted by Bromham [6], “the typical home studio is entirely unsuitable for mixing records, so there is a greater need than ever to grasp how acoustics will impact our environment and how to work around these inherent shortcomings.” Uninformed microphone placement, an unsuitable recording environment, or simply a poor performance or instrument further increases the need for an expert mix engineer [7].

In live situations, especially in small venues, the mixing task is particularly demanding and crucial, due to problems such as acoustic feedback, room resonances and poor equipment. In such cases, having a competent operator at the desk is the exception rather than the rule.

These observations, described in further detail in [114], indicate that there is a clear need for systems that take care of the mixing stage of music production for live and recording situations. By obtaining a high-quality mix quickly and autonomously, home recording becomes more affordable, smaller music venues are freed from the need for expert operators for their front of house and monitor systems, and musicians can increase their productivity and focus on the creative aspects of music production.

Professional audio engineers are often under pressure to produce high-quality content quickly and at low cost [8]. While they may be unlikely to relinquish control entirely to autonomous mix software, assistance with tedious, time-consuming tasks would be highly beneficial. This can be implemented via more powerful, intelligent, responsive, intuitive algorithms and interfaces [9].

Throughout the history of technology, innovation has been met with resistance and skepticism, in particular from professional users who fear seeing their roles disrupted or made obsolete. Music production technology may be especially susceptible to this kind of opposition, as it is characterized by a tendency towards nostalgia [6,10], and it is concerned with aesthetic value in addition to technical excellence and efficiency. The introduction of artificial intelligence into the field of audio engineering has prompted an outcry from practitioners who reject the concept of previously manual components of their jobs being automated [11].

However, the evolution of music is intrinsically linked to the development of new instruments and tools, and essentially utilitarian inventions such as automatic vocal riding, drum machines, electromechanical keyboards and digital pitch correction have been famously used and abused for creative effect. These advancements have changed the nature of the sound engineering profession from primarily technical to increasingly expressive [12]. Generally, there is economic, technological and artistic merit in exploiting the immense computing power and flexibility that today’s digital technology affords, to venture away from the rigid structure of the traditional music production toolset. As more and more industries are starting to consider what implications AI may have (or is already having) for those who work in it, it is opportune to describe the state of the art and offer suggestions about which new tools may become part of the sound engineer’s arsenal.

Rapid growth in the quantity of unprocessed audio material has resulted in a similar growth in the engineerin...

Table of contents

• Cover
• Endorsement
• Series Information
• Half Title
• Title Page
• Copyright Page
• Contents
• Figures
• Tables
• Preface
• Acknowledgments
• Part I What Do We Already Know?
• Part II How Do We Construct Intelligent Structures?
• Part III How Do We Perform Intelligent MusicProduction?
• Appendix A Additional Resources
• Bibliography
• Index