Exercises & setup
Every chapter ends with exercises. Pen-and-paper and concept ones live on the page; coding ones are small Python projects with tests.
Kinds of exercise
- Concept: answer in a sentence or two. Checks you understood the idea, not the formula.
- Pen & paper: short derivations. This is where the math becomes yours.
- Explore the demo: change something in the chapter’s interactive demo and explain what happens.
- Code: fill in the
TODOs in a Python file until the tests pass.
Each exercise is marked ●○○ (warm-up), ●●○ (core) or ●●● (challenge). Open the hint before the solution. Tick “I’ve done this one” to track your progress (it is saved in this browser only).
One-time setup for the coding labs
You need Python 3.10 or newer.
# 1. Get all labs at once (or download one chapter's zip from its page)
curl -LO https://rl.xhelio.ai/exercises/rl-exercises.zip
unzip rl-exercises.zip && cd rl-exercises
# 2. Make an environment and install the few dependencies
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt # numpy, matplotlib, pytest, torch (CPU is fine)Parts I and II need only numpy and matplotlib. torch is used from Chapter 6 on; every lab runs on a laptop CPU in a few minutes at most.
Working on a lab
cd 05-td-learning
cat README.md # what to build, in what order
python -m pytest -q # tests fail until you fill in the TODOs
python -m pytest -q -k sarsa # run just one group of tests
RL_SOLUTIONS=1 python -m pytest -q # check the reference solutions (all pass)Each lab folder has the same layout:
| Path | What it is |
|---|---|
README.md | The tasks, in order, and what you should see when you are done |
*.py | Your files: functions with TODO and raise NotImplementedError |
test_*.py | Tests. They import your file, or the solution when RL_SOLUTIONS=1 |
solutions/ | Reference solutions. Try not to peek until the tests pass or you are truly stuck |
Many labs also have a run.py or plot.py that trains the agent you built and saves a plot, so you can see it learn.