A self-paced course · 21 chapters · exercises with solutions

Reinforcement learning,
from scratch to LLMs

Start with a robot in a grid and a row of slot machines. End with how GRPO, verifiable rewards and agentic RL post-train today's language models. Every chapter has a demo you can play with, exercises to work through (pen and paper, and code with tests), and solutions you can open when you're ready.

Start with chapter 1 →Jump to RL for LLMsSet up the exercises

Part I

Foundations

Agents, rewards, values. Everything here fits in a table — no neural networks yet.

  1. 01What is reinforcement learning?Agent, environment, action, reward. How learning from reward differs from learning from labels.Being written
  2. 02Bandits: exploration vs. exploitationThe one-state RL problem. ε-greedy, UCB, Thompson sampling, and regret.Being written
  3. 03Markov decision processesStates, transitions, returns, discounting, policies, and value functions.Being written
  4. 04Bellman equations and dynamic programmingSolving an MDP when you know the rules: policy evaluation, policy iteration, value iteration.Being written
  5. 05Learning from experience: Monte Carlo and TDLearning values without a model. TD(0), SARSA, Q-learning, and on- vs. off-policy.Being written

Part II

Deep RL

Neural networks as value functions and policies, and the policy-gradient family that LLM training is built on.

  1. 06Function approximation and DQNWhen the table is too big: neural Q-functions, experience replay, target networks.Being written
  2. 07Policy gradients and REINFORCEOptimize the policy directly. The log-derivative trick, baselines, and variance.Being written
  3. 08Actor-critic and advantage estimationA learned critic to cut variance. Advantages, n-step returns, and GAE.Being written
  4. 09Trust regions: TRPO and PPOTake big, safe policy steps. Importance ratios, clipping, and the algorithm behind RLHF.Being written
  5. 10The wider RL landscapeModel-based RL and AlphaZero/MuZero, offline RL, exploration, and where each idea resurfaces in LLMs.Being written

Part III

RL for LLMs

How RL turns a pretrained language model into an assistant and a reasoner: RLHF, DPO, GRPO, verifiable rewards, agents, and the current frontier.

  1. 11The language model as a policyTokens as actions, prompts as states. Pretraining → SFT → RL, and why a KL leash to the reference model.Being written
  2. 12RLHF: reward models and PPOThe InstructGPT recipe: preference data, a Bradley–Terry reward model, and PPO with a KL penalty.Being written
  3. 13Direct preference optimizationSkip the reward model: derive DPO from the KL-regularized objective, and meet its variants.Being written
  4. 14Where rewards come fromHuman labels, AI feedback, Constitutional AI, LLM judges, rubrics, and verifiers.Being written
  5. 15RL with verifiable rewards: reasoning models and GRPOo1 and DeepSeek-R1. Group-relative advantages, R1-Zero, and why checkable answers changed everything.Being written
  6. 16The critic-free algorithm zooRLOO, GRPO, Dr. GRPO, DAPO, GSPO, CISPO and friends — one estimator, different knobs.Being written
  7. 17Process rewards and test-time computeRewarding steps vs. outcomes, PRMs, search, and trading inference compute for accuracy.Being written
  8. 18Agentic and multi-turn RLTool use, coding agents, environments, and credit assignment over long horizons.Being written
  9. 19Failure modes: reward hacking and friendsOver-optimization, reward hacking, length bias, sycophancy, and entropy collapse — and the fixes.Being written
  10. 20RL systems for LLMsWhy rollouts dominate cost, async and off-policy training, train–inference mismatch, and the open-source stacks.Being written
  11. 21The frontier (as of October 2026)What the latest models do in post-training, and the open problems worth watching.Being written

How to use this course

  1. Read with the demo open. Each idea comes with something to poke at. Change a slider before you read the explanation of what it does.
  2. Do the exercises before opening solutions. Pen-and-paper ones build the math; code ones come with tests so you know when you're right.
  3. Follow the arrows to Part III. Boxes marked “Where this shows up in LLMs” connect each classic idea to modern post-training.
  4. Your progress stays in this browser. Tick off exercises and chapters as you go.