Python Reinforcement Learning

Solve complex real-world problems by mastering reinforcement learning algorithms using OpenAI Gym and TensorFlow

Sudharsan Ravichandiran, Sean Saito, Rajalingappaa Shanmugamani, Yang Wenzhuo

• Publisher: Packt Publishing
• Year: 2019
• Print ISBN: 9781838649777
• eBook ISBN: 9781838640149
• Edition: 1
• Topic: Computer Science
• Subtopic: Artificial Intelligence (AI) & Semantics
• Index: Computer Science

Apply modern reinforcement learning and deep reinforcement learning methods using Python and its powerful libraries

Key Features

• Your entry point into the world of artificial intelligence using the power of Python
• An example-rich guide to master various RL and DRL algorithms
• Explore the power of modern Python libraries to gain confidence in building self-trained applications

Book Description

Reinforcement Learning (RL) is the trending and most promising branch of artificial intelligence. This Learning Path will help you master not only the basic reinforcement learning algorithms but also the advanced deep reinforcement learning algorithms.

The Learning Path starts with an introduction to RL followed by OpenAI Gym, and TensorFlow. You will then explore various RL algorithms, such as Markov Decision Process, Monte Carlo methods, and dynamic programming, including value and policy iteration. You'll also work on various datasets including image, text, and video. This example-rich guide will introduce you to deep RL algorithms, such as Dueling DQN, DRQN, A3C, PPO, and TRPO. You will gain experience in several domains, including gaming, image processing, and physical simulations. You'll explore TensorFlow and OpenAI Gym to implement algorithms that also predict stock prices, generate natural language, and even build other neural networks. You will also learn about imagination-augmented agents, learning from human preference, DQfD, HER, and many of the recent advancements in RL.

By the end of the Learning Path, you will have all the knowledge and experience needed to implement RL and deep RL in your projects, and you enter the world of artificial intelligence to solve various real-life problems.

This Learning Path includes content from the following Packt products:

• Hands-On Reinforcement Learning with Python by Sudharsan Ravichandiran
• Python Reinforcement Learning Projects by Sean Saito, Yang Wenzhuo, and Rajalingappaa Shanmugamani

What you will learn

• Train an agent to walk using OpenAI Gym and TensorFlow
• Solve multi-armed-bandit problems using various algorithms
• Build intelligent agents using the DRQN algorithm to play the Doom game
• Teach your agent to play Connect4 using AlphaGo Zero
• Defeat Atari arcade games using the value iteration method
• Discover how to deal with discrete and continuous action spaces in various environments

Who this book is for

If you're an ML/DL enthusiast interested in AI and want to explore RL and deep RL from scratch, this Learning Path is for you. Prior knowledge of linear algebra is expected.

Learning to Play Go

When considering the capabilities of AI, we often compare its performance for a particular task with what humans can achieve. AI agents are now able to surpass human-level competency in more complex tasks. In this chapter, we will build an agent that learns how to play what is considered the most complex board game of all time: Go. We will become familiar with the latest deep reinforcement learning algorithms that achieve superhuman performances, namely AlphaGo, and AlphaGo Zero, both of which were developed by Google's DeepMind. We will also learn about Monte Carlo tree search, a popular tree-searching algorithm that is an integral component of turn-based game agents.

This chapter will cover the following topics:

• Introduction to Go and relevant research in AI
• Overview of AlphaGo and AlphaGo Zero
• The Monte Carlo tree search algorithm
• Implementation of AlphaGo Zero

A brief introduction to Go

Go is a board game that was first recorded in China two millennia ago. Similar to other common board games, such as chess, shogi, and Othello, Go involves two players alternately placing black and white stones on a 19x19 board with the objective of capturing as much territory as possible by surrounding a larger total area of the board. One can capture their opponent's pieces by surrounding the opponent's pieces with their own pieces. Captured stones are removed from the board, thereby creating a void in which the opponent can no longer place stones unless the territory is captured back.

A game ends when both players refuse to place a stone or either player resigns. Upon the termination of a game, the winner is decided by counting each player's territory and the number of captured stones.

Go and other board games

Researchers have already created AI programs that outperform the best human players in board games such as chess and backgammon. In 1992, researchers from IBM developed TD-Gammon, which used classic reinforcement learning algorithms and an artificial neural network to play backgammon at the level of a top player. In 1997, Deep Blue, a chess-playing program developed by IBM and Carnegie Mellon University, defeated then world champion Garry Kasparov in a six-game face off. This was the first time that a computer program defeated the world champion in chess.

Developing Go playing agents is not a new topic, and hence one may wonder what took so long for researchers to replicate such successes in Go. The answer is simple—Go, despite its simple rules, is a far more complex game than chess. Imagine representing a board game as a tree, where each node is a snapshot of the board (which we also refer to as the board state) and its child nodes are possible moves the opponent can make. The height of the tree is essentially the number of moves a game lasts. A typical chess game lasts 80 moves, whereas a game in Go lasts 150; almost twice as long. Moreover, while the average number of possible moves in a chess turn is 35, a Go player has 250 possible plays per move. Based on these numbers, Go has 10⁷⁶¹ total possible games, compared to 10¹²⁰ games in chess. It is impossible to enumerate every possible state in Go in a computer, and the sheer complexity of the game has made it difficult for researchers to develop an agent that can play the game at a world-class level.

Go and AI research

In 2015, researchers from Google's DeepMind published a paper in Nature that detailed a novel reinforcement learning agent for Go called AlphaGo. In October of that year, AlphaGo beat Fan Hui, the European champion, 5-0. In 2016, AlphaGo challenged Lee Sedol, who, with 18 world championship titles, is considered one of the greatest players in modern history. AlphaGo won 4-1, marking a watershed moment in deep learning research and the game's history. In the following year, DeepMind published an updated version of AlphaGo, AlphaGo Zero, which defeated its predecessor 100 times in 100 games. In just a matter of days of training, AlphaGo and AlphaGo Zero were able to learn and surpass the wisdom that mankind has accumulated over the thousands of years of the game's existence.

The following sections will discuss how AlphaGo and AlphaGo Zero work, including the algorithms and techniques that they use to learn and play the game. This will be followed by an implementation of AlphaGo Zero. Our exploration begins with Monte Carlo tree search, an algorithm that is integral to both AlphaGo and AlphaGo Zero for making decisio...

Table of contents

• Title Page
• Copyright and Credits
• About Packt
• Contributors
• Preface
• Introduction to Reinforcement Learning
• Getting Started with OpenAI and TensorFlow
• The Markov Decision Process and Dynamic Programming
• Gaming with Monte Carlo Methods
• Temporal Difference Learning
• Multi-Armed Bandit Problem
• Playing Atari Games
• Atari Games with Deep Q Network
• Playing Doom with a Deep Recurrent Q Network
• The Asynchronous Advantage Actor Critic Network
• Policy Gradients and Optimization
• Balancing CartPole
• Simulating Control Tasks
• Building Virtual Worlds in Minecraft
• Learning to Play Go
• Creating a Chatbot
• Generating a Deep Learning Image Classifier
• Predicting Future Stock Prices
• Capstone Project - Car Racing Using DQN
• Looking Ahead
• Assessments
• Other Books You May Enjoy