Interactive Displays

Natural Human-Interface Technologies

Achintya K. Bhowmik

• Publisher: Wiley
• Year: 2014
• ISBN:
  9781118706206
• Edition: 1
• Topic: Technology & Engineering
• Subtopic: Electrical Engineering & Telecommunications
• Index: Technology & Engineering

How we interface and interact with computing, communications and entertainment devices is going through revolutionary changes, with natural user inputs based on touch, voice, and vision replacing or augmenting the use of traditional interfaces based on the keyboard, mouse, joysticks, etc. As a result, displays are morphing from one-way interface devices that merely show visual content to two-way interaction devices that provide more engaging and immersive experiences. This book provides an in-depth coverage of the technologies, applications, and trends in the rapidly emerging field of interactive displays enabled by natural human-interfaces.

Key features:

• Provides a definitive reference reading on all the touch technologies used in interactive displays, including their advantages, limitations, and future trends.
• Covers the fundamentals and applications of speech input, processing and recognition techniques enabling voice-based interactions.
• Offers a detailed review of the emerging vision-based sensing technologies, and user interactions using gestures of hands, body, face, and eye gazes.
• Discusses multi-modal natural user interface schemes which intuitively combine touch, voice, and vision for life-like interactions.
• Examines the requirements and technology status towards realizing "true" 3D immersive and interactive displays.

Chapter 1
Senses, Perception, and Natural Human-Interfaces for Interactive Displays

Achintya K. Bhowmik

Intel Corporation, USA

1.1 Introduction

Visual displays are now an integral part of a wide variety of electronic devices as the primary human interface to the computing, communications and entertainment systems which have become ubiquitous elements of our daily lives at home, work, or on the go. Whether it be the watches on our wrists, or the mobile phones that we are carrying everywhere with us in our pockets or purses, or the tablets that we are using for surfing the web and consuming multimedia content, or the laptop and desktop computers on which we are getting our work done, or the large-screen television sets at the center of our living rooms, or the presentation projectors in the business meetings, the visual display is the “face” of all these devices to us, the users.

The same applies to a plethora of vertical applications, such as the check-in kiosks at airports, check-out kiosks at retail stores, signages at shopping malls, public displays at museums – the list goes on and on. The wide array of applications and insatiable market demands have fuelled worldwide research and development to advance visual display technologies and products of all form factors in the past decades, ranging from mobile displays to large screens [1–5].

A quick glance at the market size helps us grasp just how pervasive visual displays have become in our lives. In the last five years, according to the display industry analysis firm IHS, the industry shipped nearly 17 billion flat-panel displays [6]. Also, to get a sense of the rate of adoption, the annual shipment of visual displays has grown more than 50% over this period.

In general, an electronic device performs three basic functions: receive instructions from the user, execute certain processing functions according to the instructions and information received, and present output or results of the processing to the user. As an example, when the author was typing this chapter on his laptop computer, he used a keyboard and a mouse to input the information, the world-processing software application executed on the microprocessor translated the keystrokes and mouse taps into the desired text and format, and the liquid-crystal display of the laptop computer displayed the text on the screen as a real-time visual representation or output. Hence, the display subsystem in such devices already plays a critical role by presenting information to the user – and, until recently, barring some exceptions, the majority of the electronic devices sported a display device whose sole function was just that – to display the visual information.

However, human-computer interaction and user interface paradigms have been undergoing a surge of innovation and rapid evolution in recent years. The ways we interact with computers had already gone through a transformation in the past decades, with the graphical user interfaces that use a mouse and keyboard as input devices replacing the old command-line interfaces that used text-based inputs. We are now witnessing the next revolution with the advent of more natural user interfaces, where the user interacts with the computing devices with touch, gesture, voice, facial expressions, eye gaze, and even thoughts!

Advanced sensors, systems, algorithms, and applications are being developed and demonstrated for natural and engaging interactions, where the computing devices understand the users' expressions and emotions in addition to the intent. These new interface technologies and the ensuing new class of applications present exciting opportunities for the display technology and the consumer electronics industry at large. With the integration of natural user interfaces, the display device morphs from a one-way interface device that merely shows visual content, to a two-way interaction device that also directly receives user inputs and thus enables interactive applications and immersive experiences. The proliferation of touch-screens and touch-optimized interfaces and applications has already brought this transformation to mobile displays, and now the adoption of an extended array of natural interfaces promise to redefine the whole spectrum of displays and systems by making them more interactive.

This book presents a comprehensive treatment of the natural human interface technologies and applications that are enabling the emergence of highly interactive displays and systems. So, what are “interactive displays”? We define them to be the displays that not only show visual information on the screens, but also sense and understand human actions and receive direct user inputs. Equipped with human-like sensing and perception technologies, a “truly” interactive display will “feel” and detect our touch, “hear” and respond to our voice, “see” and recognize our faces and facial expressions, “understand” and interpret our gestural instructions conveyed by the movement of the hands and fingers or other body parts, and even “infer” our intent based on the context.

While these goals may seem rather ambitious, as the examples shown in Figure 1.1 illustrate, systems of various form factors and applications with natural user interaction technologies are already making a large impact in the market by offering easy and intuitive human interfaces. As reviewed throughout the book, significant advances are taking place in natural sensing and inference technologies as well as system integration and application developments, which are expected to bring about new frontiers in interactive displays.

Figure 1.1 Interactive displays and systems of a wide range of form factors and applications are already gaining a large foothold in the market, and some examples are shown above. The displays in many of these systems assume a new role of direct human-interface device, besides the traditional role of displaying visual information to the user.

The block diagram shown in Figure 1.2 depicts the generic functional modules and flow of an interactive display system. The interactions between the user and the display system are orchestrated by the interfaces, namely the input and the output blocks shown in the beginning and at the end. The input block consists of sensors that transform the physical stimuli resulting from user inputs into electrical signals, while the output block performs the reverse function of providing system responses to user actions in the form of physical stimuli that the users can sense and perceive. The blocks in between perform the necessary signal processing and computing functions to facilitate these interactions.

Figure 1.2 Functional block diagram of an interactive display system. The input and the output blocks orchestrate the interactions between the user and the display, while the signal processing and computing functions facilitate these interactions.

In this chapter, we first review the basics of human sensing and perception, especially the mechanisms and processes that we deploy in our day-to-day interactions with the physical world. Building on this, we then provide an overview of human-computer interactions utilizing natural interface technologies based on touch-, voice-, and vision-based sensing and interactions, following a brief review of the most successful legacy interfaces. The subsequent chapters delve into the details of each of these input and interaction modalities, providing in-depth discussion on the fundamentals of the technologies and their applications in human interface schemes, as well as combinations of them towards realizing intuitive multisensory and multimodal interactions. The book concludes with a chapter on the fundamental requirements, technology development status, and outlook towards realizing “true” 3D interactive displays that would provide lifelike immersive interaction experiences.

1.2 Human Senses and Perception

We start with the assertion that the ultimate goal of implementing a human-device interface scheme is to make the interaction experience natural, intuitive, and immersive for the user. While the limitations of the technologies at hand require the designers and engineers to make compromises and settle for a subset of these goals for specific product implementations, we continue to make advances towards realizing this overarching objective.

Let us elaborate on this a little. By natural, we mean using our natural faculties for communication and interaction with the devices. We use multisensory and multimodal interface schemes to comprehend our surroundings and communicate with each other in our daily lives, seamlessly combining multiple interaction modalities such as voice, facial expressions, eye gaze, hand and body gestures, touch, smell and taste. The addition of natural interfaces can thus bring lifelike experiences to human-device interactions.

By intuitive, we refer to interfaces that require minimal (ideally no) training for the user to engage and interact with the devices, taking advantage of the years of training that we have already gone through in dealing with the world while growing up!

By immersive, we allude to an experience where the border between the real world and the virtual world is blurred, with the computers or devices becoming extensions of our body and brain to aid us in...

Table of contents

• Cover
• Wiley-SID Series in Display Technology
• Title Page
• Copyright
• About the Author
• List of Contributors
• Series Editor's Foreword
• Preface
• List of Acronyms
• Chapter 1: Senses, Perception, and Natural Human-Interfaces for Interactive Displays
• Chapter 2: Touch Sensing
• Chapter 3: Voice in the User Interface
• Chapter 4: Visual Sensing and Gesture Interactions
• Chapter 5: Real-Time 3D Sensing With Structured Light Techniques
• Chapter 6: Real-Time Stereo 3D Imaging Techniques
• Chapter 7: Time-of-Flight 3D-Imaging Techniques
• Chapter 8: Eye Gaze Tracking
• Chapter 9: Multimodal Input for Perceptual User Interfaces
• Chapter 10: Multimodal Interaction in Biometrics: Technological and Usability Challenges
• Chapter 11: Towards “True” 3D Interactive Displays
• Index
• End User License Agreement