Apache Mesos Cookbook

David Blomquist, Tomasz Janiszewski

• Publisher: Packt Publishing
• Year: 2017
• ISBN:
  9781785880933
• Edition: 1
• Topic: Computer Science
• Subtopic: Databases
• Index: Computer Science

Over 50 recipes on the core features of Apache Mesos and running big data frameworks in MesosAbout This Book• Learn to install and configure Mesos to suit the needs of your organization• Follow step-by-step instructions to deploy application frameworks on top of Mesos, saving you many hours of research and trial and error• Use this practical guide packed with powerful recipes to implement Mesos and easily integrate it with other application frameworksWho This Book Is ForThis book is for system administrators, engineers, and big data programmers. Basic experience with big data technologies such as Hadoop or Spark would be useful but is not essential. A working knowledge of Apache Mesos is expected.What You Will Learn• Set up Mesos on different operating systems• Use the Marathon and Chronos frameworks to manage multiple applications• Work with Mesos and Docker• Integrate Mesos with Spark and other big data frameworks• Use networking features in Mesos for effective communication between containers• Configure Mesos for high availability using Zookeeper• Secure your Mesos clusters with SASL and Authorization ACLs• Solve everyday problems and discover the best practicesIn DetailApache Mesos is open source cluster sharing and management software. Deploying and managing scalable applications in large-scale clustered environments can be difficult, but Apache Mesos makes it easier with efficient resource isolation and sharing across application frameworks.The goal of this book is to guide you through the practical implementation of the Mesos core along with a number of Mesos supported frameworks. You will begin by installing Mesos and then learn how to configure clusters and maintain them. You will also see how to deploy a cluster in a production environment with high availability using Zookeeper.Next, you will get to grips with using Mesos, Marathon, and Docker to build and deploy a PaaS. You will see how to schedule jobs with Chronos. We'll demonstrate how to integrate Mesos with big data frameworks such as Spark, Hadoop, and Storm. Practical solutions backed with clear examples will also show you how to deploy elastic big data jobs.You will find out how to deploy a scalable continuous integration and delivery system on Mesos with Jenkins. Finally, you will configure and deploy a highly scalable distributed search engine with ElasticSearch.Throughout the course of this book, you will get to know tips and tricks along with best practices to follow when working with Mesos.Style and approachThis step-by-step guide is packed with powerful recipes on using Apache Mesos and shows its integration with containers and big data frameworks.

Understanding the Scheduler API

In this chapter, you will learn about the frameworks and how they interact with Mesos. To do this, we will develop a simple framework.

The following are the recipes covered in this chapter:

• Installing Protobuf
• Registering frameworks
• Handling events
• Declining offers
• Scheduling tasks
• Acknowledging task updates
• Killing tasks
• State persistence
• Reconciliation

Introduction

The following recipes will guide you through creating a simple Mesos framework. It's important that this framework is prepared to be as simple as possible, just to demonstrate how frameworks interact with Mesos. However, the price of simplicity is robustness. It's not designed to handle production traffic but could be used as a starting point for creating real-world solutions.

Our framework will be created in Go. Go is getting more and more popular in system infrastructure, mostly because it gives C/C++ system performance with syntax and memory management known from higher-level languages such as Python. Another point in its favor is a powerful standard library. Our framework will require only one external dependency, Protobuf: https://github.com/golang/protobuf.

In our implementation, we will use the Mesos HTTP API. It's worth mentioning that this API is by design not RESTful, but still pretty easy to use.

Before you start, install Go on your machine and create a new directory where we can keep all the framework files.

Installing Protobuf

In this recipe, we will be installing Protobuf to generate Golang structures used in communication with Mesos.

Getting ready

Install Golang, Protobuf, and the Golang Protobuf bindings:

 sudo add-apt-repository ppa:ubuntu-lxc/lxd-stable
sudo apt-get update
sudo apt-get install golang git
export GOPATH=~/go
export PATH=$PATH:$GOPATH/bin
sudo apt-get install protobuf-compiler
go get -u github.com/golang/protobuf/{proto,protoc-gen-go} 

How to do it...

1. Create a project directory and go there:

 mkdir -p $GOPATH/src/simple-scheduler
cd $GOPATH/src/simple-scheduler 

2. Download the Mesos protobuf message definitions:

 wget https://raw.githubusercontent.com/apache/mesos/170ac/include/mesos/v1/mesos.proto
wget https://raw.githubusercontent.com/apache/mesos/170ac/include/mesos/v1/scheduler/scheduler.proto 

3. Tweak them to our needs. Change the default package of generated bindings to one that matches Golang. We will keep the whole project in one main package and the generated bindings should fit in it. To make this, apply the following patches to scheduler.proto and mesos.proto:

 cat <<EOF | patch
--- mesos.proto 2016-07-28 00:39:01.644273001 +0200
+++ mesos.proto 2016-08-08 22:16:37.112678075 +0200
@@ -18,6 +18,7 @@

option java_package = "org.apache.mesos.v1";
option java_outer_classname = "Protos";
+option go_package = "main";


/**
EOF

cat <<EOF | patch
--- scheduler.proto 2016-08-08 22:16:30.372706543 +0200
+++ scheduler.proto 2016-07-28 00:39:39.244296938 +0200
@@ -14,12 +14,13 @@
// See the License for the specific language governing permissions and
// limitations under the License.

-import "mesos/v1/mesos.proto";
+import "mesos.proto";

-package mesos.v1.scheduler;
+package mesos.v1;

option java_package = "org.apache.mesos.v1.scheduler";
option java_outer_classname = "Protos";
+option go_package = "main";

/**
EOF 

4. Generate bindings for Mesos Protobuf messages:

 protoc --go_out=. *.proto 

5. As a result, the working directory should have the following content:

 mesos.pb.go mesos.proto scheduler.pb.go scheduler.proto 

6. And when we try to build the project with go build, we should get the following error:

 # github.com/janisz/mesos-cookbook/4_understanding_frameworks
runtime.main: call to external function main.main
runtime.main: main.main: not defined
runtime.main: undefined: main.main 

This is an expected error since we have only declared the data transfer object.

How it works...

Protobufs are a technology to serialize structured data. It was developed at Google to provide an easy-to-use, language-agnostic way to describe data that is consumed and produced by the system. Besides the serialization standard, it allows for validation of the structure of messages and backward compatibility of schema. It could be compared to XML, but it was designed to be smaller and easier to write and understand. Protocol buffers comes with a binary serializer as well as JSON.

Mesos uses Protobuf to specify schema for all data transfer objects. In this recipe, we compile that schema from proto files to Go language structures. Now we can use these structures to easily interact with Mesos.

Registering frameworks

In this recipe, we will learn how frameworks register in Mesos to receive offers and state updates.

How to do it...

We will create the scheduler.go file and implement our framework inside of it.

Before we start, we need to define some globals and imports that we will need later:

 import (
 "bufio"
 "bytes"
 "log"
 "net/http"
 "os"
 "strconv"
 "strings"
 "github.com/golang/protobuf/jsonpb"
)
// Url to Mesos master scheduler API
con...

Table of contents

• Title Page
• Copyright
• Credits
• About the Authors
• About the Reviewer
• www.PacktPub.com
• Customer Feedback
• Preface
• Getting Started with Apache Mesos
• Implementing High Availability with Apache ZooKeeper
• Running and Maintaining Mesos
• Understanding the Scheduler API
• Managing Containers
• Deploying PaaS with Marathon
• Job Scheduling with Metronome
• Continuous Integration with Jenkins