Reading Guide: Algorithms for Reinforcement Learning

Algorithms for Reinforcement Learning by Csaba Szepesvári is one of the most efficient books in the field: about a hundred pages that move from Markov decision processes to temporal-difference learning, Q-learning and actor-critic methods, with the theory kept in view throughout. It is short, but it is dense. This guide helps you read it in the right order and at the right speed.

Get the book: View Algorithms for Reinforcement Learning on Amazon

What this book is (and who it's for)

The book was published in 2010 in Morgan & Claypool's Synthesis Lectures on Artificial Intelligence and Machine Learning. Szepesvári, a computing science professor at the University of Alberta, focuses on reinforcement learning algorithms that build on the theory of dynamic programming. He catalogs the main learning problems, explains the core ideas, surveys a large number of algorithms that were state of the art at the time, and discusses what is known about their theoretical properties and limitations. The author's page notes that the bibliography cites more than 200 works.

It is best for:

• Graduate students and advanced undergraduates in machine learning, control or operations research.
• ML engineers who have used RL libraries and now want to understand why the algorithms work.
• Researchers who want a compact, precise reference to the classical algorithms.

It is not a gentle first introduction. If you have never seen RL before, start with a more narrative text and come back to this one.

Before you start

You will read more comfortably if you have:

• Probability: expectations, conditional probability and Markov chains.
• Linear algebra: vectors, matrices, inner products and least squares.
• Some analysis: norms and the idea of a contraction. The appendix covers Banach's fixed-point theorem, so you can learn it there if needed.
• Basic programming: a Python notebook with NumPy is enough to implement the tabular algorithms on small gridworld problems as you read.

The author's web page also lists an errata file, which is worth keeping open alongside the text.

How to read it

The book is organized into three main chapters plus a short chapter of pointers and a theory appendix. Here is a suggested roadmap.

1. Markov Decision Processes (read carefully)

This chapter sets up the vocabulary: MDPs, value functions and the dynamic programming algorithms for solving them. Everything later depends on it. Work through the definitions slowly and make sure you can write down the Bellman equations from memory.

2. The appendix (read early if the math is new)

The appendix on the theory of discounted MDPs explains contractions and the fixed-point argument behind value iteration. Reading it right after Chapter 1 makes the convergence arguments in later chapters much easier to follow. If you are already comfortable with contraction mappings, skim it.

3. Value Prediction Problems (the heart of the book)

This chapter covers estimating the value of a fixed policy. It starts with temporal-difference learning in finite state spaces (tabular TD(0), every-visit Monte Carlo, and TD(λ) as a bridge between them) and then moves to large state spaces with function approximation, gradient TD methods and least-squares methods, ending with a discussion of how to choose the function space. On a first pass, focus on the tabular section and on TD(λ) with linear function approximation; return to gradient TD and least-squares methods on a second pass.

4. Control

Here the goal shifts from evaluating a policy to finding a good one. The chapter opens with a catalog of learning problems, then covers interactive learning (online and active learning in bandits and in MDPs), direct methods such as Q-learning with and without function approximation, and actor-critic methods. Readers interested in practice can prioritize Q-learning and actor-critic; theory-minded readers will enjoy the bandit and exploration material.

5. For Further Exploration

A short chapter of further reading, applications and software. Read it last and use it as a map for what to study next.

Key ideas to take away

• RL learns from partial, delayed feedback. Unlike supervised learning, the learner sees only the consequences of its own actions, and those actions shape future data.
• Dynamic programming is the backbone. Most algorithms in the book can be read as sample-based, approximate versions of value iteration or policy iteration.
• Prediction comes before control. Learning to evaluate a policy well is the foundation for improving it.
• Function approximation changes everything. Methods that are safe in the tabular case can behave very differently with approximation, which motivates gradient TD and least-squares approaches.
• Exploration is a problem in its own right. The bandit material shows how to balance trying new actions against exploiting what is known.
• Theory tells you when to trust an algorithm. Convergence results and their limits are presented alongside each method.

Practice and discussion questions

1. Write the Bellman equation for a policy's value function and explain why value iteration converges in a discounted MDP.
2. Implement tabular TD(0) and every-visit Monte Carlo on a small random-walk problem. How do their errors compare as data grows?
3. What does the λ parameter in TD(λ) control, and what happens at λ = 0 and λ = 1?
4. Why can combining off-policy learning with function approximation cause instability, and how do gradient TD methods address it?
5. Compare Q-learning and actor-critic: what does each method learn, and when might you prefer one?
6. In a multi-armed bandit, how would you explain the exploration–exploitation trade-off to a product manager?
7. How does the choice of features in linear function approximation limit what a value-prediction method can learn?

Pair it with

• A narrative introduction: Sutton and Barto's Reinforcement Learning: An Introduction (MIT Press) gives a gentler, example-driven path through many of the same ideas and makes an excellent companion.
• Deep RL in practice: after this book, explore how neural networks are used as function approximators in modern deep RL; the foundations here make that material much easier to follow.
• Applied AI research: Advances in Soft Computing (MICAI 2021, Part II), also in our store, includes robotics and intelligent-application papers where decision-making methods come into play. Browse more on the BooksPlusAi shelf.

Final thoughts

This book repays slow, active reading. Take it one section at a time, implement the small algorithms as you go, and revisit the theory once the intuition clicks. By the end you will have a clear mental model of how classical reinforcement learning fits together, and a solid foundation for everything that has been built on it since.

Start learning: Get Algorithms for Reinforcement Learning on Amazon

As an Amazon Associate, BooksPlusAi may earn from qualifying purchases.

Sources

• Csaba Szepesvári: Algorithms of Reinforcement Learning (author's page)
• ODBMS.org: Algorithms for Reinforcement Learning
• Springer: Algorithms for Reinforcement Learning