Mastering OpenCV 3 - Second Edition

Daniel Lelis Baggio, Shervin Emami, David Millan Escriva, Khvedchenia Ievgen, Jason Saragih, Roy Shilkrot

• Publisher: Packt Publishing
• Year: 2017
• ISBN:
  9781786466563
• Edition: 2
• Topic: Computer Science
• Subtopic: Computer Vision & Pattern Recognition
• Index: Computer Science

Practical Computer Vision ProjectsAbout This Book• Updated for OpenCV 3, this book covers new features that will help you unlock the full potential of OpenCV 3• Written by a team of 7 experts, each chapter explores a new aspect of OpenCV to help you make amazing computer-vision aware applications• Project-based approach with each chapter being a complete tutorial, showing you how to apply OpenCV to solve complete problemsWho This Book Is ForThis book is for those who have a basic knowledge of OpenCV and are competent C++ programmers. You need to have an understanding of some of the more theoretical/mathematical concepts, as we move quite quickly throughout the book.What You Will Learn• Execute basic image processing operations and cartoonify an image• Build an OpenCV project natively with Raspberry Pi and cross-compile it for Raspberry Pi.text• Extend the natural feature tracking algorithm to support the tracking of multiple image targets on a video• Use OpenCV 3's new 3D visualization framework to illustrate the 3D scene geometry• Create an application for Automatic Number Plate Recognition (ANPR) using a support vector machine and Artificial Neural Networks• Train and predict pattern-recognition algorithms to decide whether an image is a number plate• Use POSIT for the six degrees of freedom head pose• Train a face recognition database using deep learning and recognize faces from that databaseIn DetailAs we become more capable of handling data in every kind, we are becoming more reliant on visual input and what we can do with those self-driving cars, face recognition, and even augmented reality applications and games. This is all powered by Computer Vision.This book will put you straight to work in creating powerful and unique computer vision applications. Each chapter is structured around a central project and deep dives into an important aspect of OpenCV such as facial recognition, image target tracking, making augmented reality applications, the 3D visualization framework, and machine learning. You'll learn how to make AI that can remember and use neural networks to help your applications learn.By the end of the book, you will have created various working prototypes with the projects in the book and will be well versed with the new features of OpenCV3.Style and approachThis book takes a project-based approach and helps you learn about the new features by putting them to work by implementing them in your own projects.

Face Recognition Using Eigenfaces or Fisherfaces

In this chapter, we cover the following:

• Face detection
• Face preprocessing
• Training a machine-learning algorithm from collected faces
• Face recognition
• Finishing touches

Introduction to face recognition and face detection

Face recognition is the process of putting a label to a known face. Just like humans learn to recognize their family, friends, and celebrities just by seeing their face, there are many techniques for a computer to learn to recognize a known face. These generally involve four main steps:

1. Face detection: This is the process of locating a face region in an image (a large rectangle near the center of the following screenshot). This step does not care who the person is, just that it is a human face.
2. Face preprocessing: This is the process of adjusting the face image to look more clear and similar to other faces (a small grayscale face in the top-center of the following screenshot).
3. Collecting and learning faces: This is the process of saving many preprocessed faces (for each person that should be recognized), and then learning how to recognize them.
4. Face recognition: This is the process that checks which of the collected people are most similar to the face in the camera (a small rectangle on the top-right of the following screenshot).

Note that the phrase face recognition is often used by the general public for finding positions of faces (that is, face detection, as described in step 1), but this book will use the formal definition of face recognition referring to step 4 and face detection referring to step 1.

The following screenshot shows the final WebcamFaceRec project, including a small rectangle at the top-right corner highlighting the recognized person. Also notice the confidence bar that is next to the preprocessed face (a small face at the top-center of the rectangle marking the face), which in this case shows roughly 70 percent confidence that it has recognized the correct person:

The current face detection techniques are quite reliable in real-world conditions, whereas current face recognition techniques are much less reliable when used in real-world conditions. For example, it is easy to find research papers showing face recognition accuracy rates above 95 percent, but when testing those same algorithms yourself, you may often find that accuracy is lower than 50 percent. This comes from the fact that current face recognition techniques are very sensitive to exact conditions in the images, such as the type of lighting, direction of lighting and shadows, exact orientation of the face, expression of the face, and the current mood of the person. If they are all kept constant when training (collecting images) as well as when testing (from the camera image), then face recognition should work well, but if the person was standing to the left-hand side of the lights in a room when training, and then stood to the right-hand side while testing with the camera, it may give quite bad results. So the dataset used for training is very important.

Face preprocessing (step 2) aims to reduce these problems, such as by making sure the face always appears to have similar brightness and contrast, and perhaps making sure the features of the face will always be in the same position (such as aligning the eyes and/or nose to certain positions). A good face preprocessing stage will help improve the reliability of the whole face recognition system, so this chapter will place some emphasis on face preprocessing methods.

Despite the big claims about face recognition for security in the media, it is unlikely that the current face recognition methods alone are reliable enough for any true security system, but they can be used for purposes that don't need high reliability, such as playing personalized music for different people entering a room or a robot that says your name when it sees you. There are also various practical extensions to face recognition, such as gender recognition, age recognition, and emotion recognition.

Step 1 - face detection

Until the year 2000, there were many different techniques used for finding faces, but all of them were either very slow, very unreliable, or both. A major change came in 2001 when Viola and Jones invented the Haar-based cascade classifier for object detection, and in 2002 when it was improved by Lienhart and Maydt. The result is an object detector that is both fast (it can detect faces in real time on a typical desktop with a VGA webcam) and reliable (it detects approximately 95 percent of frontal faces correctly). This object detector revolutionized the field of face recognition (as well as that of robotics and computer vision in general), as it finally allowed real-time face detection and face recognition, especially as Lienhart himself wrote the object detector that comes free with OpenCV! It works not only for frontal faces but also side-view faces (referred to as profile faces), eyes, mouths, noses, company logos, and many other objects.

This object detector was extended in OpenCV v2.0 to also use LBP features for detection based on work by Ahonen, Hadid, and Pietikäinen in 2006, as LBP-based detectors are potentially several times faster than Haar-based detectors, and don't have the licensing issues that many Haar detectors have.

The basic idea of the Haar-based face detector is that if you look at most frontal faces, the region with the eyes should be darker than the forehead and cheeks, and the region with the mouth should be darker than cheeks, and so on. It typically performs about 20 stages of comparisons like this to decide if it is a face or not, but it must do this at each possible position in the image and for each possible size of the face, so in fact it often does thousands of checks per image. The basic idea of the LBP-based face detector is similar to the Haar-based one, but it uses histograms of pixel intensity comparisons, such as edges, corners, and flat regions.

Rather than have a person decide which comparisons would best define a face, both Haar- and LBP-based face detectors can be automatically trained to find faces from a large set of images, with the information stored as XML files to be used later. These cascade classifier detectors are typically trained using at least 1,000 unique face images and 10,000 non-face images (for example, photos of trees, cars, and text), and the training process can take a long time even on a multi-core desktop (typically a few hours for LBP but 1week for Haar!). Luckily, OpenCV comes with some pretrained Haar and LBP detectors for you to use! In fact you can detect frontal faces, profile (side-view) faces, eyes, or noses just by loading different cascade classifier XML files to the object detector, and choose between the Haar or LBP detector, based on which XML file you choose.

Implementing face detection using OpenCV

As mentioned previously, OpenCV v2.4 comes with various, pretrained XML detectors that you can use for different purposes. The following table lists some of the most popular XML files:

Type of cascade classifier	XML filename
Face detector (default)	haarcascade_frontalface_default.xml
Face detector (fast Haar)	haarcascade_frontalface_alt2.xml
Face detector (fast LBP)	lbpcascade_frontalface.xml
Profile (side-looking) face detector	haarcascade_profileface.xml
Eye detector (separate for left and right)	haarcascade_lefteye_2splits.xml
Mouth detector	haarcascade_mcs_mouth.xml
Nose detector	haarcascade_mcs_nose.xml
Whole person detector	haarcascade_fullbody.xml

Haar-based detectors are stored in the datahaarcascades folder and LBP-based detectors are stored in the datalbpcascades folder of the OpenCV root folder, such as C:opencvdatalbpcascades.

For our face recognition project, we want to detect frontal faces, so let's use the LBP face detector because it is the fastest and doesn't have patent licensing issues. Note that this pretrained LBP face detector that comes with OpenCV v2.x is not tuned as well as the pretrained Haar face detectors, so if you want more reliable face detection then you may want to train your own LBP face detector or use a Haar face detector.

Loading a Haar or LBP detector for object or face detection

To perform object or face detection, first you must load the pretrained XML file using OpenCV's CascadeClassifier class as follows:

 CascadeClassifier faceDetector; faceDetector.load(faceCascadeFilename); 

This can load Haar or LBP detectors just by giving a different filename. A very common mistake when using this is to provide the wrong folder or filename, but depending on your build environment, the load() method will either return false or generate a C++ exception (and exit your program with an assert error). So it is best to surround the load() method with a try... catch block and display a nice error message to the user if something went wrong. Many beginners skip checking for errors, but it is crucial to show a help message to the user when something did not load correctly, otherwise you may spend a very long time debugging other parts of your code before eventually realizing something did not load. A simple error message can be displayed as follows:

 CascadeClassifier faceDetector; try { faceDetector.load(faceCascadeFilename); } catch (cv::Exception e) {} if ( faceDetector.empty() ) { cerr << "ERROR: Couldn't load Face Detector ("; cerr << faceCascadeFilename << ")!" << endl; exit(1); } 

Accessing the webcam

To grab frames from a computer's webcam or even from a video file, you can simply call the VideoCapture::open() function with the camera number or video filename, then grab the frames using the C++ stream operator, as mentioned in the section,Accessing the webcam in Chapter 1, Cartoonifier and Skin Changer for Raspberry Pi.

Detecting an object using the Haar or LBP Classifier

Now that we have loaded the classifier (just once during initialization), we can use it to detect faces in each new camera frame. But first, we should do some initial processing of the camera image just for face detection, by performing the following steps:

1. Grayscale color conversion: Face detection only works on grayscale images. So we should convert the color camera frame to grayscale.
2. Shrinking the camera image: The speed of face detection depends on the size of the input image (it is very slow for large images but fast for small images), and yet detection is still fairly reliable even at low resolutions. So we should shrink the camera image to a more reasonable size (or use a large value for minFeatureSize in the detector, as explained shortly).
3. Histogram equalization: Face detection is not as reliable in low-light conditions. So we should perform histogram equalization to improve the contrast and brightness.

Grayscale color conversion

We can easily convert an RGB color image to grayscal...

Table of contents

• Title Page
• Copyright
• Credits
• About the Authors
• About the Reviewer
• www.PacktPub.com
• Customer Feedback
• Preface
• Cartoonifier and Skin Changer for Raspberry Pi
• Exploring Structure from Motion Using OpenCV
• Number Plate Recognition using SVM and Neural Network
• Non-Rigid Face Tracking
• 3D Head Pose Estimation Using AAM and POSIT
• Face Recognition Using Eigenfaces or Fisherfaces