Text Mining with Machine Learning

Principles and Techniques

Jan Žižka, František Dařena, Arnošt Svoboda

• Publisher: CRC Press
• Year: 2019
• ISBN:
  9780429890260
• Edition: 1
• Topic: Computer Science
• Subtopic: Data Mining
• Index: Computer Science

This book provides a perspective on the application of machine learning-based methods in knowledge discovery from natural languages texts. By analysing various data sets, conclusions which are not normally evident, emerge and can be used for various purposes and applications. The book provides explanations of principles of time-proven machine learning algorithms applied in text mining together with step-by-step demonstrations of how to reveal the semantic contents in real-world datasets using the popular R-language with its implemented machine learning algorithms. The book is not only aimed at IT specialists, but is meant for a wider audience that needs to process big sets of text documents and has basic knowledge of the subject, e.g. e-mail service providers, online shoppers, librarians, etc.

The book starts with an introduction to text-based natural language data processing and its goals and problems. It focuses on machine learning, presenting various algorithms with their use and possibilities, and reviews the positives and negatives. Beginning with the initial data pre-processing, a reader can follow the steps provided in the R-language including the subsuming of various available plug-ins into the resulting software tool. A big advantage is that R also contains many libraries implementing machine learning algorithms, so a reader can concentrate on the principal target without the need to implement the details of the algorithms her- or himself. To make sense of the results, the book also provides explanations of the algorithms, which supports the final evaluation and interpretation of the results. The examples are demonstrated using realworld data from commonly accessible Internet sources.

Chapter 1

Introduction to Text Mining with Machine Learning

1.1 Introduction

In the last few years, we have seen an explosion in the amount of data as the result of many activities. The data comes from various sources and is available in many formats. This has been enabled, primarily, by the massive advance on the Internet. Various devices like smart phones enable communication and the use of applications anytime and anywhere; many activities like shopping, interacting with government institutions, or providing support to customers, are moving to digital environments; many documents are being digitalized; people meet and interact on virtual platforms. The transformation of the Web into Web 2.0 [26], where the content is actively created by users, has thrown open to the masses, many possible avenues to express their ideas, recommendations, or attitudes.

It is obvious that texts written in a natural language are a natural way of human communication. Textual documents are thus strongly related to many human activities and they have become a source that is worth analyzing. The results of such an analysis can bring useful insights in practically all domains.

As the ability of people to analyze texts remains essentially the same, the availability of more data requires new computational methods in order to find something useful in large collections of documents. Thus, a discipline known as text mining has become very popular and attractive. Text mining can be defined as a knowledge-intensive process in which a user interacts with a collection of documents by using analytic tools in order to identify and explore interesting patterns [87]. Applications can be found in marketing, competitive intelligence, banking, health care, manufacturing, security, natural sciences, and many other domains [254, 192].

Computers are able to analyze only the syntactic aspect of texts, which means that they are able to recognize how words are arranged in the documents. Because texts in a natural language are written using a grammar, some syntactic patterns in a text can be more or less easily identified. Semantics refers to the meaning of a word or group of words in a context. Without perfect understanding of a language it is not possible to completely understand the meaning. Fortunately, is is possible to solve many practical problems even without the full understanding of a text because syntax and semantics are often closely related. If two texts use the same words and syntactic structures, they are likely to be semantically similar and can, for example, belong to the same class of documents [196].

Documents can generally be analyzed in two different ways. The statistical or machine learning approach uses mathematical representation of texts. Linguistic methods, using natural language processing techniques, represent texts using language models where the meaning and different relationships are contained. Text mining uses both approaches to find knowledge, usually in a large number of texts [254, 271].

1.2 Relation of Text Mining to Data Mining

Text mining encompasses a wide variety of tasks that can bring information about different aspects of texts. The typical text mining tasks include [196, 279, 123]:

• ■ categorization of documents – assigning a document to one or more predefined category (e.g., assigning a newspaper article to one or more categories, labeling e-mails as spam or ham);
• ■ clustering – grouping documents according to their similarity, for example, in order to identify documents sharing a common topic;
• ■ summarization – finding the most important parts in one or more documents and creating a text that is significantly shorter than the original;
• ■ information retrieval – retrieving documents that match a query representing information needed from a large collection of documents;
• ■ extracting the meaning of documents or their parts – identifying hidden topics, analyzing sentiment, opinion, or emotions;
• ■ information extraction – extracting structured information like entities, events, or relations from unstructured texts;
• ■ association mining – finding associations between concepts or terms in texts;
• ■ trend analysis – looking at how concepts contained in documents change in time;
• ■ machine translation – converting a text written in one language to a text in another language.

Some of the text mining tasks are very similar to the tasks of data mining. Data mining is the automatic or semiautomatic process of finding implicit, previously unknown, and potentially useful knowledge in collections of electronically stored data. The knowledge has a form of structural patterns in data that can be also used to make predictions or provide answers in the future [280].

Data mining includes many different methods, tools, algorithms, or models. All of them require the data to be in a structured form. This means that the data can be represented in a tabular form as in a relational database. The data takes the form of a set of examples (or instances, data points, observations) described by specific values of their features (or attributes, variables, fields).

The features can be of several types [75, 174]:

• ■ categorical (nominal) – the domain is a discrete set of values where ordering does not make sense;
• ■ binary – a special type of categorical attribute with only two possible values;
• ■ ordinal – the domain is a discrete set of values that can be ordered;
• ■ numerical – the value of a feature is a number, either integer or continuous.

An example of a structured representation of data describing purchases in a retail store is in Table 1.1. Every purchase is characterized by a customer (described by age, education level, and gender), date of purchase, and the total price.

Table 1.1: A structured representation of purchases in a retail store.

Age	Gender	Education	Date	Price
35	female	primary	2019-02-10	20.0
40	male	tertiary	2019-02-14	28.4
21	male	secondary	2019-01-30	15.1
63	female	secondary	2019-03-01	11.9

Such a format is, however, not typical for texts. Generally, a text is a string written in a natural language consisting of parts (words) with certain meaning that are combined according to some rules (syntax). The texts can also be of a different range. A unit of a text can be a sentence, a few sentences combined in a paragraph, or much longer texts, like web pages, e-mails, articles, or books. Sometimes, a text can be just a few words that are not a valid sentence, which is quite typical, for example, for short posts on social networks.

In order to be able to apply data mining methods to texts, they need to be converted to a structured representation. A classic structured representation of texts as vectors in a vector space model is known as bag-of-words and the process of inferring this representation is described in chapter 3. Another, more modern, representation that is based on embedding words to a continuous vectors space referred as to word embeddings is discussed in Chapter 13.

There are a few problems related to the bag-of-words representation, which are not that typical for data mining tasks in general [134, 65, 188, 170, 202]:

• ■ Independently of the complexity of features, the input space for text mining problems is large. It is not uncommon that the dictionaries of various natural languages contain hundreds of thousands of words. When considering not only words as features but also, for example, combinations, mutual positions, or grammatical relationships between words, the complexity further increases. Of course, not all words of a language will appear in most d...

Table of contents

• Cover
• Title Page
• Copyright Page
• Dedication Page
• Preface
• Table of Contents
• Authors’ Biographies
• 1. Introduction to Text Mining with Machine Learning
• 2. Introduction to R
• 3. Structured Text Representations
• 4. Classification
• 5. Bayes Classifier
• 6. Nearest Neighbors
• 7. Decision Trees
• 8. Random Forest
• 9. Adaboost
• 10. Support Vector Machines
• 11. Deep Learning
• 12. Clustering
• 13. Word Embeddings
• 14. Feature Selection
• References
• Index