New Realities in Audio

A Practical Guide for VR, AR, MR and 360 Video

Stephan Schütze, Anna Irwin-Schütze

• Publisher: CRC Press
• Year: 2018
• ISBN:
  9781351727433
• Edition: 1
• Topic: Computer Science
• Subtopic: Programming Games
• Index: Computer Science

The new realities are here. Virtual and Augmented realities and 360 video technologies are rapidly entering our homes and office spaces. Good quality audio has always been important to the user experience, but in the new realities, it is more than important, it's essential. If the audio doesn't work, the immersion of the experience fails and the cracks in the new reality start to show.

This practical guide helps you navigate the challenges and pitfalls of designing audio for these new realities. This technology is different from anything we've seen before and requires an entirely new approach; this book will introduce the broad concepts you need to know before delving into the practical detail you need.

Key Features

• This book covers audio for all types of new reality technology. At the moment, VR and 360 video are getting a lot of press, but in a few years we will be hearing a lot more about Augmented and Mixed reality technologies as well.

• A practical guide to creating, designing and implementing audio for this new technology by a leading sound design and implementation expert.

• Conceptual explanations address the new approaches necessary to designing effective audio for the new realities.

• Real-world examples and analysis of what does and does not work including detailed case study discussions.

1

new realities

introduction

Beginnings are important. Beginnings are when we establish trust and define much of how a relationship will progress. The new formats of virtual reality (VR), alternate reality (AR), and mixed reality (MR) are at a beginning. These new “realities” are at a delicate stage and I sincerely believe they will define a new era of communications and entertainment. There is a risk, however, that if they are not presented in the right way or form the right relationships, they could easily just become another tech fad, another grab for quick cash from some people wanting to take advantage of unwitting consumers looking for the next big thing. If this happens then these new formats are going to fail.

Let me begin, therefore, by stating I do not consider myself an expert in the new realities, not yet at least. I have spent the past few years exploring, discovering, and researching possibilities. I have worked for several leading companies specializing in the new realities and have participated in some very exciting projects. I am also incredibly passionate about the creative potential offered by these new formats. I believe that we have opened up an area that is so new to all of us, and potentially so very different to everything we have used up till now, that it would be unrealistic for anyone to claim to be an “expert” just yet.

I am quite sincere when I say I do not think we currently have the slightest idea of the true potential of where the new realities can take us. But I do think they are an entirely new approach to communication. Our current media started with the still photograph hundreds of years ago; then through technology and brilliant minds we developed moving pictures (film) then radio added sound to the mix. Then we developed television, video games, and mobile devices. If we consider all of those to be evolutionary states along the same line, then I believe that the new reality is not the next step along that line.

While the new technologies owe much to our previous creative media, I think we are going to discover that they are the start of an entirely new branch of human communication and engagement. This is why it is so important that we develop the new realities carefully to allow them to reach their real potential. I also believe that we cannot even conceive of the types of experiences that will exist in 5–10 years’ time. This both excites and terrifies me.

I have heard people refer to this early stage of the new realities as “the Wild West of technology,” a time of discovery, of vague laws and rules where the shape of things is not yet determined. An opportunity for heroes to carve out reputations and become legends. This analogy works well, especially when you remember the Wild West was also a time of snake oil salesmen; people who made wild claims about their expertise and the wondrous abilities of their products. This is what we need to guard against the most if our use of new realities is to reach its true potential.

There is a conflict in this book that has been there since Day 1. My position has shifted back and forth as I have done research, as I have experienced different VR and AR projects and as I have drafted sections of this text. I think in many ways this conflict might be the subtext of the whole book because I think it is currently important to the evolution of this new format. The conflict is the science versus the creative process.

As you progress through this book, you will discover many references to the science and math that drives this technology. I am constantly amazed at the genius minds that allow us to work in virtual environments and create incredible and engaging visual and audio aspects of our experiences. The science is critical and core to this technology. However, I am also constantly amazed at how often very basic, age-old techniques such as those we have been using in theater for hundreds of years are not just still relevant, but in some ways, they are more relevant than ever.

The science allows us to flex our creative muscles and provide tools that enable us to create, but again and again I see that so many of our old methods of narration, presentation, and communication continue to serve us no matter how advanced our technology becomes. So there is conflict in this book as I cite the amazing capabilities of head-related transfer function (HRFT) spatial audio tools and at the same time suggest we adopt old-fashioned stagecraft to get the best results for our projects. There is conflict as we explore just how immersive a VR experience with amazing audio can be, but discover that there is a significant level of smoke and mirrors, that is, just as critical to the engagement as the technology that drives the delivery platforms. I do not think this conflict is a bad thing, in fact I think a mix of bleeding edge technology and established narrative techniques may be the best approach for many things, but I do think it is worthy of mention as this conflict is something that has come up again and again as I have progressed in writing this book.

In this book, I want to start a dialogue about what audio for VR, AR, and MR currently looks like, what it can be in the near future and what ultimately we might achieve when we have had time to become experts. So let me be your guide and suggest ways in which you can explore this Wild West frontier together.

I feel that sound is half the experience. Filmmakers should focus on making sure the soundtracks are really the best they can possibly be because in terms of an investment, sound is where you get the most bang for your buck

George Lucas 2004, Mixonline

George Lucas has consistently spoken about how important the sound is to a quality film. Game audio folk love to share this quote, to use it to give us more legitimacy. It helps us feel like we have some value. We tell developers, good quality audio is “nice” and will benefit their product.

For the new realities, the quality of the audio is not simply beneficial to your end result, or a “nice to have,” it is absolutely critical. A VR/MR/AR experience will live or die based on how well you use and implement the audio. We will return to this point repeatedly throughout this book, as there are many different aspects of the new realities that reinforce this simple truth.

why this book

Because of the nature of the new technologies, I have invited others in the field to contribute their thoughts and approaches to working with VR/AR/MR. Some of these people have significant experience working with the new technologies and all of these people are brilliant at what they do. Their contribution to this industry and to this book is significant because we all benefit from the sharing of experience. Throughout this book you will find contributions from various creative minds writing about aspects of the new realities they are passionate about, and how they approached various challenges to producing content. It is important to me to acknowledge the people who are helping to build this industry through their hard work and clever thoughts.

As part of researching this book, I spoke with many of my industry colleagues in the games industry. I asked in particular about the use of obstruction and occlusion in video game projects. The technology to support these two features has existed for several years, but they are still not utilized as often as they could be. When I queried my colleagues I got a range of reasons. Often the computational resources needed for accurate occlusion were too high but one statement I felt cut right to the heart of the matter:

The trick for us (resource wise) was to create a ‘difference’ rather than go for absolute realism.

I have heard this comment or similar from many audio professionals. Our job is not to simulate reality for the sake of accuracy. Often, simulating reality is the first step and then from there we alter the content to make it more fun or more entertaining. Hollywood films are an excellent example of the world being significantly exaggerated from the audio point of view. So if a character moves from an outside space into a room we would apply some reverb. The change is what is important, the exact value of the reverb echoes and if they accurately match the room space exactly is not only less important, it probably borders on irrelevant to most of our audience.

The new realities will certainly need to utilize some important math and physics to assist in the creation of convincing and immersive content; there are brilliant engineers doing just that right now. The work they do with the math and science of manipulating audio allow those of us who work on the creative side to accomplish amazing results. But I am not an engineer and this book is aimed at composers and sound designers, artists, and creative producers. I will trust in the engineers to create amazing tools, and instead use this book to discuss our approach to getting the most out of them.

The advances in computer technology have been incredible over the last 20 years and graphics rendering has improved remarkably, but we can still usually tell the difference between CG and real-world footage. The same applies to audio creation. We can spatialize audio and utilize some incredible technology to create immersive experiences, but we are still many years from being able to simulate the real world precisely. Because of this there will be a continuing need to balance out the technology we have with techniques we use to produce convincing and exciting sonic content.

Right now, technology is advancing almost faster than we can keep up. In the last few weeks, as I write this book, there have been multiple spatial audio toolsets released and it is almost impossible to learn them all. We have more options than we have had before, but the craft of clever sound design is as important as ever and this book will discuss and compare the careful balancing act we must perform between these two elements.

The art of Foley is as important as ever.

There are so many aspects of creative media that are critical to its success and the new realities, if anything, add more challenges for the creator. But there are some core truths about creative content and why it is relevant to our audiences. Throughout this book we will investigate and discuss many of them. Viktor Phoenix provides a wonderful insight and overview into one important aspect of the new realities: agency for the audience.

Agents of Change

Creating Presence through Real-Time Spatial Audio

Viktor Phoenix

All Great Stories Are about Change

Narrative elements move from one state to another. Gameplay state changes as players progress through games. Plots move forward, characters change their points of view, bison is hunted across cave walls, points are scored, battles are won, wars are lost, and heroes make journeys.

If a Story Doesn’t Change, It’s Dead in Its Tracks

There are many forms that those changes can take.

Time Domain

In interactive experiences, one way in which change happens is in the time domain—events transpire at different intervals during each playthrough and users can unlock narrative elements in a nonlinear order.

Frequency Domain

Another way that change happens is in the frequency domain—events that can be repeated might happen only once or they might happen once every 2 seconds.

Spatial Domain

In immersive experiences, change can also happen in the spatial domain—events and objects move in multiple directions and, when users have freedom to move through 3D space, at any distance.

Free to Be Me or Not

Allowing players to drive change is a large part of the appeal of video games. Applying that same level of interactivity to nongaming immersive experiences, to give users the freedom to decide how they can be a part of the narrative and give them the control they crave to truly be a part of the story, is a large part of determining the success of an immersive exp...

Table of contents

• Cover
• Title Page
• Copyright Page
• Table of Contents
• acknowledgments
• authors
• contributors
• photography
• 1 new realities
• 2 definitions and core concepts
• 3 why isn’t this easy? The many challenges of the new realities
• 4 what we’re doing now: analysis of existing projects
• 5 creation and implementation: putting it into practice
• 6 sound? what sound? gathering assets for your project
• 7 music: how and why music in the new realties forces a shift in perspective
• 8 working together: navigating the team project minefield
• 9 platform considerations: how different platforms affect what you can, can’t, or shouldn’t do
• 10 roadmap for the future: what do we do next?
• index