Hands-On Intelligent Agents with OpenAI Gym

Your guide to developing AI agents using deep reinforcement learning

Praveen Palanisamy

• Publisher: Packt Publishing
• Year: 2018
• Print ISBN:
  9781788836579
• Edition: 1
• eBook ISBN:
  9781788835138
• Topic: Computer Science
• Subtopic: Artificial Intelligence (AI) & Semantics
• Index: Computer Science

Implement intelligent agents using PyTorch to solve classic AI problems, play console games like Atari, and perform tasks such as autonomous driving using the CARLA driving simulator

Key Features

• Explore the OpenAI Gym toolkit and interface to use over 700 learning tasks
• Implement agents to solve simple to complex AI problems
• Study learning environments and discover how to create your own

Book Description

Many real-world problems can be broken down into tasks that require a series of decisions to be made or actions to be taken. The ability to solve such tasks without a machine being programmed requires a machine to be artificially intelligent and capable of learning to adapt. This book is an easy-to-follow guide to implementing learning algorithms for machine software agents in order to solve discrete or continuous sequential decision making and control tasks.

Hands-On Intelligent Agents with OpenAI Gym takes you through the process of building intelligent agent algorithms using deep reinforcement learning starting from the implementation of the building blocks for configuring, training, logging, visualizing, testing, and monitoring the agent. You will walk through the process of building intelligent agents from scratch to perform a variety of tasks. In the closing chapters, the book provides an overview of the latest learning environments and learning algorithms, along with pointers to more resources that will help you take your deep reinforcement learning skills to the next level.

What you will learn

• Explore intelligent agents and learning environments
• Understand the basics of RL and deep RL
• Get started with OpenAI Gym and PyTorch for deep reinforcement learning
• Discover deep Q learning agents to solve discrete optimal control tasks
• Create custom learning environments for real-world problems
• Apply a deep actor-critic agent to drive a car autonomously in CARLA
• Use the latest learning environments and algorithms to upgrade your intelligent agent development skills

Who this book is for

If you're a student, game/machine learning developer, or AI enthusiast looking to get started with building intelligent agents and algorithms to solve a variety of problems with the OpenAI Gym interface, this book is for you. You will also find this book useful if you want to learn how to build deep reinforcement learning-based agents to solve problems in your domain of interest. Though the book covers all the basic concepts that you need to know, some working knowledge of Python programming language will help you get the most out of it.

Implementing an Intelligent - Autonomous Car Driving Agent using Deep Actor-Critic Algorithm

In Chapter 6, Implementing an Intelligent Agent for Optimal Control using Deep Q-Learning, we implemented agents using deep Q-learning to solve discrete control tasks that involve discrete actions or decisions to be made. We saw how they can be trained to play video games such as Atari, just like we do: by looking at the game screen and pressing the buttons on the game pad/joystick. We can use such agents to pick the best choice given a finite set of choices, make decisions, or perform actions where the number of possible decisions or actions is finite and typically small. There are numerous real-world problems that can be solved with an agent that can learn to take optimal through to discrete actions. We saw some examples in Chapter 6, Implementing an Intelligent Agent for Optimal Discrete Control using Deep Q-Learning.

In the real world, there are other classes of problems and tasks that require lower-level actions to be performed that are continuous values and not discrete. For example, an intelligent temperature control system or a thermostat needs to be capable of making fine adjustments to the internal control circuits to maintain a room at the specified temperature. The control action signal may include a continuous valued real number (such as 1.456) to control heating, ventilation, and air conditioning (HVAC) systems. Consider another example in which we want to develop an intelligent agent to drive a car autonomously. Humans drive a car by shifting gears, pressing the accelerator or brake pedal, and steering the car. While the current gear is going to be one of a possible set of five to six values, depending on the transmission system of the car, if an intelligent software agent has to perform all of those actions, it has to be able to produce continuous valued real numbers for the throttle (accelerator), braking (brake), and steering.

In cases like these examples, where we need the agent to take continuous valued actions, we can use policy gradient-based actor-critic methods to directly learn and update the agent's policy in the policy space, rather than through a state and/or action value function like in the deep Q-learning agent we saw in Chapter 6, Implementing an Intelligent Agent for Optimal Discrete Control using Deep Q-Learning. In this chapter, we will start from the basics of an actor-critic algorithm and build our agent gradually, while training it to solve various classic control problems using OpenAI Gym environments along the way. We will build our agent all the way up to being able to drive a car in the CARLA driving simulation environment using the custom Gym interface that we implemented in the previous chapter.

The deep n-step advantage actor-critic algorithm

In our deep Q-learner-based intelligent agent implementation, we used a deep neural network as the function approximator to represent the action-value function. The agent then used the action-value function to come up with a policy based on the value function. In particular, we used the

-greedy algorithm in our implementation. So, we understand that ultimately the agent has to know what actions are good to take given an observation/state. Instead of parametrizing or approximating a state/action action function and then deriving a policy based on that function, can we not parametrize the policy directly? Yes we can! That is the exact idea behind policy gradient methods.

In the following subsections, we will briefly look at policy gradient-based learning methods and then transition to actor-critic methods that combine and make use of both value-based and policy-based learning. We will then look at some of the extensions to the actor-critic method that have been shown to improve learning performance.

Policy gradients

In policy gradientbased methods, the policy is represented, for example, by using a neural network with parameters

, and the goal is to find the best set of parameters

. This can be intuitively seen as an optimization problem where we are trying to optimize the objective of the policy to find the best-performing policy. What is the objective of the agent's policy ? We know that the agent should achieve maximum rewards in the long term, in order to complet...

Table of contents

• Title Page
• Copyright and Credits
• Dedication
• Packt Upsell
• Contributors
• Preface
• Introduction to Intelligent Agents and Learning Environments
• Reinforcement Learning and Deep Reinforcement Learning
• Getting Started with OpenAI Gym and Deep Reinforcement Learning
• Exploring the Gym and its Features
• Implementing your First Learning Agent - Solving the Mountain Car problem
• Implementing an Intelligent Agent for Optimal Control using Deep Q-Learning
• Creating Custom OpenAI Gym Environments - CARLA Driving Simulator
• Implementing an Intelligent - Autonomous Car Driving Agent using Deep Actor-Critic Algorithm
• Exploring the Learning Environment Landscape - Roboschool, Gym-Retro, StarCraft-II, DeepMindLab
• Exploring the Learning Algorithm Landscape - DDPG (Actor-Critic), PPO (Policy-Gradient), Rainbow (Value-Based)
• Other Books You May Enjoy