Part 1. Search meets deep learning
Setting up search engines to effectively react to usersâ needs isnât an easy task. Traditionally, many manual tweaks and adjustments had to be made to a search engineâs internals to get it to work decently on a real collection of data. On the other hand, deep neural networks are very good at learning useful information about vast amounts of data. In this first part of the book, weâll start looking into how a search engine can be used in conjunction with a neural network to get around some common limitations and provide users with a better search experience.
Chapter 1. Neural search
This chapter covers
- A gentle introduction to search fundamentals
- Important problems in search
- Why neural networks can help search engines be more effective
Suppose you want to learn something about the latest research breakthroughs in artificial intelligence. What will you do to find information? How much time and work does it take to get the facts youâre looking for? If youâre in a (huge) library, you can ask the librarian what books are available on the topic, and they will probably point you to a few they know about. Ideally, the librarian will suggest particular chapters to browse in each book.
That sounds easy enough. But the librarian generally comes from a different context than you do, meaning you and the librarian may have different opinions about whatâs significant. The library could have books in various languages, or the librarian might speak a different language. Their information about the topic could be outdated, given that latest is a fairly relative point in time, and you donât know when the librarian last read anything about artificial intelligence, or if the library regularly receives publications in the field. Additionally, the librarian may not understand your inquiry properly. The librarian may think youâre talking about intelligence from the psychology perspective,[1] requiring a few iterations back and forth before you understand one another and get to the pieces of information you need.
1
This happened to me in real life.
Then, after all this, you might discover the library doesnât have the book you need; or the information may be spread among several books, and you have to read them all. Exhausting!
Unless youâre a librarian yourself, this is what often happens nowadays when you search for something on the internet. Although we can think of the internet as a single huge library, there are many different librarians out there to help you find the information you need: search engines. Some search engines are experts in certain topics; others know only a subset of a library, or only a single book.
Now imagine that someone, letâs call him Robbie, who already knows about both the library and its visitors, can help you communicate with the librarian in order to better find what youâre looking for. That will help you get your answers more quickly. Robbie can help the librarian understand a visitorâs inquiry by providing additional context, for example. Robbie knows what the visitor usually reads about, so he skips all the books about psychology. Also, having read a lot of the books in the library, Robbie has better insight into whatâs important in the field of artificial intelligence. It would be extremely helpful to have advisors like Robbie to help search engines work better and faster, and help users get more useful information.
This book is about using techniques from a machine learning field called deep learning (DL) to build models and algorithms that can influence the behavior of search engines, to make them more effective. Deep learning algorithms will play the role of Robbie, helping the search engine to provide a better search experience and to deliver more precise answers to end users.
One important thing to note is that DL isnât the same as artificial intelligence (AI). As you can see in figure 1.1, AI is a huge research field; machine learning is only part of it, and DL, in turn, is a sub-area of machine learning. Basically, DL studies how to make machines âlearnâ using the deep neural network computing model.
Figure 1.1. Artificial intelligence, machine learning, and deep learning
1.1. Neural networks and deep learning
The goal of this book is to enable you to use deep learning in the context of search engines, to improve the search experience and results. Even if youâre not going to build the next Google search, you should be able to learn enough to use DL techniques within small or medium-sized search engines to provide a better experience to users. Neural search should help you automate work that youâd otherwise have to perform manually. For example, youâll learn how to automate extraction of synonyms from search engine data, avoiding manual editing of synonym files (chapter 2). This saves time while improving search effectiveness, regardless of the specific use case or domain. The same is true for having good related-content suggestions (chapter 6). In many cases, users are satisfied with a combination of plain search together with the ability to navigate related content. Weâll also cover some more-specific use cases, such as searching content in multiple languages (chapter 7) and searching for images (chapter 8).
The only requirement for the techniques weâll discuss is that they have enough data to feed into neural networks. But itâs difficult to define the boundaries of âenough dataâ in a generic way. Letâs instead summarize the minimum number of documents (text, images, and so on) that are generally needed for each of the problems addressed in the book: see table 1.1.
Table 1.1. Per-task requirements for neural search techniques
Task | Minimum number of docs (range) | Chapter |
Learning word representations | 1,000â10,000 | 2, 5 |
Text generation | 10,000â100,000 | 3, 4 |
Learning document representations | 1,000â10,000 | 6 |
Machine translation | 10,000â100,000 | 7 |
Learning image representations | 10,000â100,000 | 8 |
Note that table 1.1 isnât meant to be strictly adhered to; these are numbers drawn from experience. For example, even if a search engine counts fewer than 10,000 documents, you can still try to implement the neural machine translation techniques from chapter 7; but you should take into account that it may be harder to get high-quality results (for example, perfect translations).
As you read the book, youâll learn a lot about DL as well as all the required search fundamentals to implement these DL principles in a search engine. So if youâre a search engineer or a programmer willing to learn neural search, this book is for you.
You arenât expected to know what DL is or how it works, at this point. Youâll get to know more as we look at some specific algorithms one by one, when they become useful for solving particular types of search problems. For now, Iâll start you off with some basic definitions. Deep learning is a field of machine learning where computers are capable of learning to represent and recognize things incrementally, by using deep neural networks. Deep artificial neural networks are a computational paradigm originally inspired by the way the brain is organized into graphs of neurons (although the brain is much more complex than an artificial neural network). Usually, information flows into neurons in an input layer, then through a network of hidden neurons (forming one or more hidden layers), and then out through neurons in an output layer. Neural networks can also be thought of as black boxes: smart functions that can transform inputs into outputs, based on what each network has been trained for. A common neural network has at least one input layer, one hidden layer, and one output layer. When a network has more than one hidden layer, we call the network deep. In figure 1.2, you can see a deep neural network with two hidden layers.
Figure 1.2. A deep neural network with two hidden layers
Before going into more detail about neural networks, letâs take a step back. I said deep learning is a subfield of machine learning, which is part of the broader area of artificial intelligence. But what is machine learning?
1.2. What is machine learning?
An overview of basic machine learning concepts is useful here before diving into DL and search specifics. Many of the concepts that apply to learning with artificial neural networks, such as supervised and unsupervised learning, training, and predicting, come from machine le...