Deep Learning Neural Networks

Design and Case Studies

Daniel Graupe

• Editore: WSPC
• Anno: 2016
• ISBN:
  9789813146471
• Argomento: Ciencia de la computación
• Categoria: Desarrollo de software

Deep Learning Neural Networks is the fastest growing field in machine learning. It serves as a powerful computational tool for solving prediction, decision, diagnosis, detection and decision problems based on a well-defined computational architecture. It has been successfully applied to a broad field of applications ranging from computer security, speech recognition, image and video recognition to industrial fault detection, medical diagnostics and finance.

This comprehensive textbook is the first in the new emerging field. Numerous case studies are succinctly demonstrated in the text. It is intended for use as a one-semester graduate-level university text and as a textbook for research and development establishments in industry, medicine and financial research.

Contents:

• Acknowledgements
• Preface
• Deep Learning Neural Networks: Methodology and Scope
• Basic Concepts of Neural Networks
• Back Propagation
• The Cognitron and Neocognitron
• Deep Learning Convolutional Neural Networks
• LAMSTAR-1 and LAMSTAR-2 Neural Networks
• Other Neural Networks for Deep Learning
• Case Studies
• Concluding Comments
• Problems
• Appendices to Case Studies of Chapter 8
• Author Index
• Subject Index

Readership: Researchers, academics, professionals, graduate and undergraduate students in machine learning, artificial intelligence, neural networks/networking, software engineering, and in their applications in medicine, security engineering and financial engineering.

CHAPTER 1

Deep Learning Neural Networks: Methodology and Scope

1.1.Definition

Deep Learning neural networks (DLNNs) may be defined as neural networks architectures that can facilitate deep learning, retrieval and analysis of data that is deeply buried in input information and not otherwise easily retrievable. Their ability to dig deeply in the input data is often superior and/or faster to other (non-neural-network) computational methods due to their efficient integration of several and often many mathematical, logical and computational methods, linear, nonlinear, analytic or heuristic, deterministic or stochastic, for a given task.

Another definition of deep learning neural networks is that, [Dong and Yu, 2014] “DLNNs are a class of machine learning techniques that exploit many layers of nonlinear information processing for supervised and unsupervised feature extraction and transformation and for pattern analysis and classification”. Also, DLNN networks are usually feedforward networks in their design.

By its name, Deep Learning is needed when simple methods are insufficient, where one must dig deep. This usually requires a “heavy arsenal” of knowledge. This “heavy” arsenal must be of varied, though strong tools. These tools must, however, be intelligently integrated. Integration must not be biased, but depend on unbiased learning by results.

Artificial neural networks can learn. Deep learning neural networks must learn and adaptively rank a whole arsenal by learning. This is their purpose, and this is what the present text is about.

Like other neural network architectures, DLNN architectures generally attempt to imitate the architecture of the biological brain to a certain degree, sometimes more and sometimes less. Their integration of algorithms that certainly do not reside in the biological brain is not too dis-similar from the way that the brain itself receives inputs from pre-processors from outside pre-processors. Light inputs are pre-processed in the retina and sound inputs are pre-processed in the cochlea (for color discrimination or sound frequency discrimination, respectively). Similarly, chemical pre-processing of odors or taste is performed in the nose or in the tongue, before being sent to the central nervous system (CNS). In some way, one may even consider literature that one reads, scientific or otherwise, as a pre-processor of knowledge.

1.2.Brief History of DNN and of its Applications

Deep Learning was one of the main goals of machine intelligence from its very beginning. It was therefore also one of the main purposes of artificial neural networks. The hope was that artificial neural networks can utilize the speed electronic computers and their related programming power to dig deeper into information than man can and that it can integrate various mathematical methods and directly apply them to data. Unveiling of the non-obvious which may however, be important to a specific application was always the expectation of scientific advancement and electronic computers were considered to be a tool for achieving it. Furthermore, it was hoped that the basis for fulfilling this purpose might be sought in a machine based on imitating the general architecture of the human brain, namely, in an artificial neural network architecture.

The first artificial neural network that was designed to be general enough for deep learning was the Back-Propagation (BP) Neural Network, proposed in 1986 by David Rumelhart et al. [Rumelhart, 1986]. (A similar design was already proposed by Paul Werbos in his 1974 Ph.D. thesis in [Werbos P J, 1974] and then by D B Parker in 1982 [Parker D B, 1982]). Back Propagation (BP) is based on Richard Bellman’s Dynamic Programming theory [Bellman, 1961], and is still employed in several major Deep Learning neural network architectures. However, despite its generality, in itself it is too slow and cannot efficiently integrate many pre-filtering or pre-processing mathematical algorithms as may be needed for deep learning.

In 1975, Kunihiko Fukushima [Fukushima K, 1975] proposed the Cognitron Neural Network to imitate the functioning of the retina for machine visual pattern recognition. He extended Cognitron in 1980, proposing the Neocognitron [Fukushima K, 1980], which was still very cumbersome and rather slow, and which, like its predecessor Cognitron, was still limited to visual pattern recognition. It was not a deep-learning network, and although it was also not a convolutional network, it later served as a basis for the most important convolutional neural networks to be discussed in Chapter 5.

The convolutional neural networks (CNN) became the most recognized and most popular deep learning neural networks. Historically, the CNN network was inspired by modelling of the visual cortex [Fukushima et al., 1980]. It originated with the work by Yann LeCun and his associates, which was concerned with image (zip-code) recognition [LeCun et al., 1989]. Therefore, it is hardly surprising that till today CNN was mainly applied to image-related problems.

In this 1989 work, the LeCun et al. incorporated convolution in their 5-layer BP-based design, thus achieving deeper and faster learning than afforded by BP alone. Though the training time for this early design was approximately 3 days, today’s CNN designs based on LeCun’s Le-Net 5 [LeCun et al., 1998] take only minutes to train (depending on the complexity of the problem involved), especially if parallel processing is employed.

Hinton and his co-workers extended the range of applications of the CNN-based architecture to speech recognition and natural language processing problems [Hinton et al., 2012]. Thus, CNN soon became the leading approach for use in (still and video) image processing and in speech processing, overshadowing other architectures, such as those based on Support Vector Machine (SVM) or on other algorithms in most such problems. Presently, the range of CNN applications spreads to many other applications, as long as these can be represented or reformulated into a 2D or higher-dimensional spatial form, namely, into matrix or tensor notation or any other suitable feature-map. Therefore, CNN became the most widely used neural network for solving complex deep learning problems.

Among the many applications of CNN that appear in the literature, we mention just a few, as follows (in addition to those mentioned earlier):

Still images and Video applications: LeCun’s application that launched CNN [LeCun et al., 1089], Ciresan’s record breaking application to handwritten text [Ciresan, 2012], the 3D application by Ji et al. [Ji, 2012] and Simonyan and Zisserman’s application to video [Simonyan, Zisserman, 2014]. Of the applications to speech we mention [Abdel Hamid et al., 2013].

Of the other fields of applications are applications to fault detection [Calderon-Martinez et al., 2006]; finance [Dixon et al., 2015]; search engines [Rios and Kavuluru, 2015], Medicine [Wallach et al., 2015] and many others in many more areas. Also see Chapter 5 and the Case Studies in Chapter 8.

In 1996 Graupe and H Kordylewski proposed a design for the Large Memory Storage and Retrieval Neural Network (LAMSTAR, or LNN) deep-learning network of unrestricted number of layers [Graupe and Kordylewski, 1996]. This neural network (NN) was developed to serve as generalized NN-based learning machine for computing prediction, detection and operational decisions from varied data sources. Data can be deterministic, stochastic, spatial, temporal, logic, time-varying, non-analytic, or a combination of the above. The LAMSTAR is a Hebbian [Hebb, 1949] NN. It follows a 1969 machine intelligence model [Graupe and Lynn, 1969] which was inspired by Emanuel Kant’s concept of “Verbindungen” (“interconnections”) in Understanding [Kant E, 1781] and by the neuronal interconnections between different cortexes and layers of the brain. Its computational power is due to its ability to integrate and rank parameters from carious co-processors, stochastic, analytic or non-analytic, including entropy, wavelets and beyond. Its speed is derived from employing the Hebbian-Pavlovian principle, Kohonen’s winner-takes-all principle [Kohonen, 1984] and from the ease that it lends itself to parallel computing.

The LAMSTAR neural network was successfully applied to problems ranging from medical prediction and diagnosis [Nigam, 2004], [Waxman et al., 2010], financial computing [Dong F, et al., 2012] to video processing [Girado et al., 2004], computer intrusion [Venkatachalam V, 2007] and to speech [Graupe and Abon, 2003]. Also see Chapter 6 and the Case Studies in Chapter 8.

The basic LAMSTAR structure (LAMSTAR 1, or LNN-1) was normalized in 2008 by Schneider and Graupe [Schneider N, Graupe D, 2008] to yield a modified LAMSTAR version (LAMSTAR-2, or LNN-2). This resulted in considerably improved performance with no effect on computational speed.

1.3.The Scope of the Present Text

Despite the short history of Deep Learning Neural Networks (DLNN), several different architectures have been proposed for it. Even within the different methodologies used in these architectures, programming an algorithm for a given problem is often a very major task. Furthermore, whereas Neural Networks are supposed to follow or to approximate a general architecture based on the organization of the biological Central Nervous System (CNS), many if not most DLNN architectures have little in common with any CNS architecture. Also, our knowledge of the CNN itself is still too weak to allow modeling of its deep learning. Many DLNN designs borrow from the broad range of mathematical techniques and fit these into a network-like algorithm. Rigid limits on architecture are therefore often too restrictive for such integration in any but simple cases, deep learning, by its definition, requires “all the tools possible”. Hence, there is no other way but to borrow from any advances in mathematics that we can. After all, mathematical knowledge, as we know it, is available to us through someone’s human brain. The price lies in the degree of convenience in integrating it all.

This leads us to divide deep leaning neural networks into three classes:

(a)	DLNNs where integration is smooth and intelligent, and where the network is computationally fasts and of a wide range of applicability;
(b)	DLNNs whose design incorporates or is based on only specific tools, to allow deep learning for only specific classes of problems, even if, in some cases, in a speedy manner;
(c)	DLNNs where integration is complex and the networks is slow and therefore of limited appeal.

Deep learning can also be achieved with non-NN (non-neural-network) architectures, such as SVM (Support Vector Machines) which are general in range of applications, but usually slow (see Case Studies in Chapter 8), especially in very complex problems, or as ad-hoc algorithms for very specific problems.

This book focuses mainly on DLNNs of class (a) above. Indeed, we intend to show that these DLNN’s are capable to deliver excellent performance in reasonable speed to a wide range of problems that require deep learning, which is their goal. Not only is their design very fast due to their NN (neural network) architecture, even allowing for integrating external mathematical and algorithmic tools, but their performance and speed is competitive with ad-hoc algorithms whose design is obviously very time consuming. Deep learnin...

Indice dei contenuti

• Cover Page
• Title
• Copyright
• Acknowledgements
• Preface
• Contents
• Chapter 1 Deep Learning Neural Networks: Methodology and Scope
• Chapter 2 Basic Concepts of Neural Networks
• Chapter 3 Back-Propagation
• Chapter 4 The Cognitron and Neocognitron
• Chapter 5 Deep Learning Convolutional Neural Networks
• Chapter 6 LAMSTAR-1 and LAMSTAR-2 Neural Networks
• Chapter 7 Other Neural Networks for Deep Learning
• Chapter 8 Case Studies
• Chapter 9 Concluding Comments
• Problems
• Appendices to Case Studies of Chapter 8
• Author Index
• Subject Index