Building Computer Vision Projects with OpenCV 4 and C++

Implement complex computer vision algorithms and explore deep learning and face detection

David Millán Escrivá, Prateek Joshi, Vinícius G. Mendonça, Roy Shilkrot

• Publisher: Packt Publishing
• Year: 2019
• ISBN:
  9781838641269
• Edition: 1
• Topic: Computer Science
• Subtopic: Computer Vision & Pattern Recognition
• Index: Computer Science

Delve into practical computer vision and image processing projects and get up to speed with advanced object detection techniques and machine learning algorithms

Key Features

• Discover best practices for engineering and maintaining OpenCV projects
• Explore important deep learning tools for image classification
• Understand basic image matrix formats and filters

Book Description

OpenCV is one of the best open source libraries available and can help you focus on constructing complete projects on image processing, motion detection, and image segmentation.

This Learning Path is your guide to understanding OpenCV concepts and algorithms through real-world examples and activities. Through various projects, you'll also discover how to use complex computer vision and machine learning algorithms and face detection to extract the maximum amount of information from images and videos. In later chapters, you'll learn to enhance your videos and images with optical flow analysis and background subtraction. Sections in the Learning Path will help you get to grips with text segmentation and recognition, in addition to guiding you through the basics of the new and improved deep learning modules. By the end of this Learning Path, you will have mastered commonly used computer vision techniques to build OpenCV projects from scratch. This Learning Path includes content from the following Packt books:

• Mastering OpenCV 4 - Third Edition by Roy Shilkrot and David Millán Escrivá
• Learn OpenCV 4 By Building Projects - Second Edition by David Millán Escrivá, Vinícius G. Mendonça, and Prateek Joshi

What you will learn

• Stay up-to-date with algorithmic design approaches for complex computer vision tasks
• Work with OpenCV's most up-to-date API through various projects
• Understand 3D scene reconstruction and Structure from Motion (SfM)
• Study camera calibration and overlay augmented reality (AR) using the ArUco module
• Create CMake scripts to compile your C++ application
• Explore segmentation and feature extraction techniques
• Remove backgrounds from static scenes to identify moving objects for surveillance
• Work with new OpenCV functions to detect and recognize text with Tesseract

Who this book is for

If you are a software developer with a basic understanding of computer vision and image processing and want to develop interesting computer vision applications with OpenCV, this Learning Path is for you. Prior knowledge of C++ and familiarity with mathematical concepts will help you better understand the concepts in this Learning Path.

Face Detection and Recognition with the DNN Module

In this chapter, we are going to learn the main techniques of face detection and recognition. Face detection is the process whereby faces are located in a whole image. In this chapter, we are going to cover different techniques to detect faces in images, from classic algorithms using cascade classifiers with Haar features to newer techniques using deep learning. Face recognition is the process of identifying a person that appears in an image. We are going to cover the following topics in this chapter:

• Face detection with different methods
• Face preprocessing
• Training a machine learning algorithm from collected faces
• Face recognition
• Finishing touches

Introduction to face detection and face recognition

Face recognition is the process of putting a label to a known face. Just like humans learn to recognize their family, friends, and celebrities just by seeing their face, there are many techniques for recognize a face in computer vision.

These generally involve four main steps, defined as follows:

1. Face detection: This is the process of locating a face region in an image (the large rectangle near the center of the following screenshot). This step does not care who the person is, just that it is a human face.
2. Face preprocessing: This is the process of adjusting the face image to look clearer and similar to other faces (a small grayscale face in the top center of the following screenshot).
3. Collecting and learning faces: This is a process of saving many preprocessed faces (for each person that should be recognized), and then learning how to recognize them.
4. Face recognition: This is the process that checks which of the collected people are most similar to the face in the camera (a small rectangle in the top right of the following screenshot).

Note that the phrase face recognition is often used by the general public to refer to finding the positions of faces (that is, face detection, as described in step 1), but this book will use the formal definition of face recognition referring to step 4, and face detection referring to Step 1.

The following screenshot shows the final WebcamFaceRec project, including a small rectangle at the top-right corner highlighting the recognized person. Also, notice the confidence bar that is next to the preprocessed face (a small face at the top center of the rectangle marking the face), which in this case shows roughly 70 percent confidence that it has recognized the correct person:

Current face detection techniques are quite reliable in real-world conditions, whereas current face recognition techniques are much less reliable when used in real-world conditions. For example, it is easy to find research papers showing face recognition accuracy rates above 95 percent, but when testing those same algorithms yourself, you may often find that accuracy is lower than 50 percent. This comes from the fact that current face recognition techniques are very sensitive to exact conditions in images, such as the type of lighting, direction of lighting and shadows, exact orientation of the face, expression of the face, and the current mood of the person. If they are all kept constant when training (collecting images), as well as when testing (from the camera image), then face recognition should work well, but if the person was standing to the left-hand side of the lights in a room when training, and then stood to the right-hand side while testing with the camera, it may give quite bad results. So, the dataset used for training is very important.

Face preprocessing aims to reduce these problems by making sure the face always appears to have similar brightness and contrast, and perhaps making sure the features of the face will always be in the same position (such as aligning the eyes and/or nose to certain positions). A good face preprocessing stage will help improve the reliability of the whole face recognition system, so this chapter will place some emphasis on face preprocessing methods.

Despite the big claims about using face recognition for security in the media, it is unlikely that the current face recognition methods alone are reliable enough for any true security system. However, they can be used for purposes that don't need high reliability, such as playing personalized music for different people entering a room, or a robot that says your name when it sees you. There are also various practical extensions to face recognition, such as gender recognition, age recognition, and emotion recognition.

Face detection

Until the year 2000, there were many different techniques used for finding faces, but all of them were either very slow, very unreliable, or both. A major change came in 2001 when Viola and Jones invented the Haar-based cascade classifier for object detection, and in 2002 when it was improved by Lienhart and Maydt. The result is an object detector that is both fast (it can detect faces in real time on a typical desktop with a VGA webcam) and reliable (it detects approximately 95 percent of frontal faces correctly). This object detector revolutionized the field of face recognition (as well as that of robotics and computer vision in general), as it finally allowed real-time face detection and face recognition, especially as Lienhart himself wrote the object detector that comes free with OpenCV! It works not only for frontal faces but also side-view faces (referred to as profile faces), eyes, mouths, noses, company logos, and many other objects.

This object detector was extended in OpenCV v2.0 to also use LBP features for detection based on the work done by Ahonen, Hadid, and Pietikäinen in 2006, as LBP-based detectors are potentially several times faster than Haar-based detectors, and don't have the licensing issues that many Haar detectors have.

OpenCV has implemented deep learning from v3.4 and it's more stable in v4.0. In this chapter, we will show how to use Single Shot Multibox Detector (SSD) algorithm for face detection.

The basic idea of the Haar-based face detector is that if you look at most frontal faces, the region with the eyes should be darker than the forehead and cheeks, the region with the mouth should be darker than the cheeks, and so on. It typically performs about 20 stages of comparisons like this to decide whether it is a face or not, but it must do this at each possible position in the image, and for each possible size of the face, so in fact, it often does thousands of checks per image. The basic idea of the LBP-based face detector is similar to the Haar-based one, but it uses histograms of pixel intensity comparisons, such as edges, corners, and flat regions.

Rather than have a person decide which comparisons would best define a face, both Haar and LBP-based face detectors can be automatically trained to find faces from a large set of images, with the information stored as XML files to be used later. These cascade classifier detectors are typically trained using at least 1,000 unique face images and 10,000 non-face images (for example, photos of trees, cars, and text), and the training process can take a long time even on a multi-core desktop (typically a few hours for LBP, but one week for Haar!). Luckily, OpenCV comes with some pretrained Haar and LBP detectors for you to use! In fact, you can detect frontal faces, profile (side-view) faces, eyes, or noses just by loading different cascade classifier XML files into the object detector and choosing between the Haar and LBP detector, based on which XML file you choose.

Implementing face detection using OpenCV cascade classifiers

As mentioned previously, OpenCV v2.4 comes with various pretrained XML detectors that you can use for different purposes. The following table lists some of the most popular XML files:

Type of cascade classifier	XML filename
Face detector (default)	haarcascade_frontalface_default.xml
Face detector (fast Haar)	haarcascade_frontalface_alt2.xml
Face detector (fast LBP)	lbpcascade_frontalface.xml
Profile (side-looking) face detector	haarcascade_profileface.xml
Eye detector (separate for left and right)	haarcascade_lefteye_2splits.xml
Mouth detector	haarcascade_mcs_mouth.xml
Nose detector	haarcascade_mcs_nose.xml
Whole person detector	haarcascade...

Table of contents

• Title Page
• Copyright and Credits
• About Packt
• Contributors
• Preface
• Getting Started with OpenCV
• An Introduction to the Basics of OpenCV
• Learning Graphical User Interfaces
• Delving into Histogram and Filters
• Automated Optical Inspection, Object Segmentation, and Detection
• Learning Object Classification
• Detecting Face Parts and Overlaying Masks
• Video Surveillance, Background Modeling, and Morphological Operations
• Learning Object Tracking
• Developing Segmentation Algorithms for Text Recognition
• Text Recognition with Tesseract
• Deep Learning with OpenCV
• Cartoonifier and Skin Color Analysis on the RaspberryPi
• Explore Structure from Motion with the SfM Module
• Face Landmark and Pose with the Face Module
• Number Plate Recognition with Deep Convolutional Networks
• Face Detection and Recognition with the DNN Module
• Android Camera Calibration and AR Using the ArUco Module
• iOS Panoramas with the Stitching Module
• Finding the Best OpenCV Algorithm for the Job
• Avoiding Common Pitfalls in OpenCV
• Other Books You May Enjoy