Big Data Analytics with Applications in Insider Threat Detection

Bhavani Thuraisingham, Pallabi Parveen, Mohammad Mehedy Masud, Latifur Khan

• Publisher: Auerbach Publications
• Year: 2017
• ISBN:
  9781351645768
• Edition: 1
• Topic: Informatique
• Subtopic: Bases de données

Today's malware mutates randomly to avoid detection, but reactively adaptive malware is more intelligent, learning and adapting to new computer defenses on the fly. Using the same algorithms that antivirus software uses to detect viruses, reactively adaptive malware deploys those algorithms to outwit antivirus defenses and to go undetected. This book provides details of the tools, the types of malware the tools will detect, implementation of the tools in a cloud computing framework and the applications for insider threat detection.

1

Introduction

1.1 Overview

The U.S. Bureau of Labor and Statistics (BLS) defines big data as a collection of large datasets that cannot be analyzed with normal statistical methods. The datasets can represent numerical, textual, and multimedia data. Big data is popularly defined in terms of five Vs: volume, velocity, variety, veracity, and value. Big data management and analytics (BDMA) requires handling huge volumes of data, both structured and unstructured, arriving at high velocity. By harnessing big data, we can achieve breakthroughs in several key areas such as cyber security and healthcare, resulting in increased productivity and profitability. Big data spans several important fields: business, e-commerce, finance, government, healthcare, social networking, and telecommunications, as well as several scientific fields such as atmospheric and biological sciences. BDMA is evolving into a field called data science that not only includes BDMA, but also machine learning, statistical methods, high-performance computing, and data management.

Data scientists aggregate, process, analyze, and visualize big data in order to derive useful insights. BLS projected both computer programmers and statisticians to have high employment growth during 2012–2022. Other sources have reported that by 2018, the United States alone could face a shortage of 140,000–190,000 skilled data scientists. The demand for data science experts is on the rise as the roles and responsibilities of a data scientist are steadily taking shape. Currently, there is no debate on the fact that data science skillsets are not developing proportionately with high industry demands. Therefore, it is imperative to bring data science research, development, and education efforts into the mainstream of computer science. Data are being collected by every organization regardless of whether it is industry, academia, or government. Organizations want to analyze this data to give them a competitive edge. Therefore, the demand for data scientists including those with expertise in BDMA techniques is growing by several folds every year.

While BDMA is evolving into data science with significant progress over the past 5 years, big data security and privacy (BDSP) is becoming a critical need. With the recent emergence of the quantified self (QS) movement, personal data collected by wearable devices and smartphone apps are being analyzed to guide users in improving their health or personal life habits. This data are also being shared with other service providers (e.g., retailers) using cloud-based services, offering potential benefits to users (e.g., information about health products). But such data collection and sharing are often being carried out without the users’ knowledge, bringing grave danger that the personal data may be used for improper purposes. Privacy violations could easily get out of control if data collectors could aggregate financial and health-related data with tweets, Facebook activity, and purchase patterns. In addition, access to the massive amounts of data collected has to be stored. Yet few tools and techniques exist for privacy protection in QS applications or controlling access to the data.

While securing big data and ensuring the privacy of individuals are crucial tasks, BDMA techniques can be used to solve security problems. For example, an organization can outsource activities such as identity management, email filtering, and intrusion detection to the cloud. This is because massive amounts of data are being collected for such applications and this data has to be analyzed. Cloud data management is just one example of big data management. The question is: how can the developments in BDMA be used to solve cyber security problems? These problems include malware detection, insider threat detection, intrusion detection, and spam filtering.

We have written this book to elaborate on some of the challenges in BDMA and BDSP as well as to provide some details of our ongoing efforts on big data analytics and its applications in cyber security. The specific BDMA techniques we will focus on include stream data analytics. Also, the specific cyber security applications we will discuss include insider threat detection. We will also describe some of the experimental systems we have designed relating to BDMA and BDSP as well as provide some of our views on the next steps including developing infrastructures for BDMA and BDSP to support education and experimentation.

This chapter details the organization of this book. The organization of this chapter is as follows. Supporting technologies for BDMA and BDSP will be discussed in Section 1.2. Our research and experimental work in stream data analytics including processing of massive data streams is discussed in Section 1.3. Application of stream data analytics to insider threat detection is discussed in Section 1.4. Some of the experimental systems we have designed and developed in topics related to BDMA and BDSP will be discussed in Section 1.5. The next steps, including developing education and experimental programs in BDMA and BDSP as well as some emerging topics such as Internet of things (IoT) security as it relates to BDMA and BDSP are discussed in Section 1.6. Organization of this book will be given in Section 1.7. We conclude this chapter with useful resources in Section 1.8. It should be noted that the contents of Sections 1.2 through 1.5 will be elaborated in Parts I through V of this book. Figure 1.1 illustrates the contents covered in this chapter.

Figure 1.1 Concepts of this chapter.

1.2 Supporting Technologies

We will discuss several supporting technologies for BDMA and BDSP. These include data security and privacy, data mining, data mining for security applications, cloud computing and semantic web, data mining and insider threat detection, and BDMA technologies. Figure 1.2 illustrates the supporting technologies discussed in this book.

Figure 1.2 Supporting technologies.

With respect to data security and privacy, we will describe database security issues, security policy enforcement, access control, and authorization models for database systems, as well as data privacy issues. With respect to data mining, which we will also refer to as data analytics, we will introduce the concept and provide an overview of the various data mining techniques to lay the foundations for some of the techniques to be discussed in Parts II through V. With respect to data mining applications in security, we will provide an overview of how some of the data mining techniques discussed may be applied for cyber security applications. With respect to cloud computing and semantic web, we will provide some of the key points including cloud data management and technologies such as resource description framework for representing and managing large amounts of data. With respect to data mining and insider threat detection, we will discuss some of our work on applying data mining for insider threat detection that will provide the foundations for the concepts to be discussed in Parts II and III. Finally, with respect to BDMA technologies, we will discuss infrastructures and frameworks, data management, and data analytics systems that will be applied throughout the various sections in this book.

1.3 Stream Data Analytics

Data streams are continuous flows of data being generated from various computing machines such as clients and servers in networks, sensors, call centers, and so on. Analyzing these data streams has become critical for many applications including for network data, financial data, and sensor data. However, mining these ever-growing data is a big challenge to the data mining community. First, data streams are assumed to have infinite length. It is impractical to store and use all the historical data for learning, as it would require an infinite amount of storage and learning time. Therefore, traditional classification algorithms that require several passes over the training data are not directly applicable to data streams. Second, data streams observe concept drift whi...

Table of contents

• Cover
• Half Title
• Title Page
• Copyright Page
• Dedication
• Contents
• Preface
• Acknowledgments
• Permissions
• Authors
• Chapter 1: Introduction
• Part I: Supporting Technologies for BDMA and BDSP
• Part II: Stream Data Analytics
• Part III: Stream Data Analytics for Insider Threat Detection
• Part IV: Experimental BDMA and BDSP Systems
• Part V: Next Steps for BDMA and BDSP
• Chapter 36: Summary and Directions
• Appendix A: Data Management Systems: Developments and Trends
• Appendix B: Database Management Systems
• Index