Notation

One set of symbols for the whole course, from gridworlds to GRPO. When a paper uses different letters, the chapter says so.

Classic RL (Parts I–II)

Symbol	Meaning
$t$ , $T$	Time step; final step of an episode (horizon)
$s_t \in \mathcal{S}$	State at time $t$ ; $\mathcal{S}$ is the set of states
$a_t \in \mathcal{A}$	Action at time $t$ ; $\mathcal{A}$ is the set of actions
$r_t$	Reward received after taking $a_t$ in $s_t$
$P(s' \mid s, a)$	Transition probability: chance of landing in $s'$
$r(s, a)$	Expected reward for taking $a$ in $s$
$\gamma \in [0, 1]$	Discount factor
$G_t = \sum_{k=0}^{T-t-1} \gamma^k r_{t+k}$	Return: discounted sum of rewards from $t$ on
$\tau = (s_0, a_0, r_0, s_1, \dots)$	A trajectory (episode)
$\pi(a \mid s)$	Policy: probability of action $a$ in state $s$
$\pi_\theta$	A policy with parameters $\theta$ (e.g. a neural network)
$V^\pi(s) = \mathbb{E}_\pi[G_t \mid s_t = s]$	State-value function
$Q^\pi(s, a) = \mathbb{E}_\pi[G_t \mid s_t = s, a_t = a]$	Action-value function
$A^\pi(s, a) = Q^\pi(s, a) - V^\pi(s)$	Advantage: how much better $a$ is than average
$V^, Q^, \pi^*$	Optimal value functions and an optimal policy
$\delta_t = r_t + \gamma V(s_{t+1}) - V(s_t)$	TD error
$\alpha$	Step size (learning rate)
$\epsilon$	Exploration rate in ε-greedy; also the clip range in PPO/GRPO (context makes it clear)
$\lambda$	Trace / GAE parameter
$J(\theta) = \mathbb{E}_{\tau \sim \pi_\theta}[G_0]$	Objective: expected return of the policy
$\rho_t(\theta) = \dfrac{\pi_\theta(a_t \mid s_t)}{\pi_{\theta_\text{old}}(a_t \mid s_t)}$	Importance ratio (PPO’s paper calls it $r_t(\theta)$ ; we use $\rho$ to avoid a clash with reward)
$\hat{A}_t$	An estimate of the advantage at step $t$

Bandits (Chapter 2)

Symbol	Meaning
$K$	Number of arms
$q_*(a)$	True mean reward of arm $a$
$Q_t(a)$	Our estimate of $q_*(a)$ at time $t$
$N_t(a)$	Times arm $a$ was pulled before time $t$
$\text{Regret}_T = T\,\max_a q_(a) - \sum_{t=1}^{T} q_(a_t)$	Reward lost by not always pulling the best arm

RL for LLMs (Part III)

Symbol	Meaning
$x \sim \mathcal{D}$	A prompt drawn from the prompt dataset
$y = (y_1, \dots, y_{\lvert y\rvert})$	A response: a sequence of tokens
$y_{<t}$	The tokens before position $t$
$\pi_\theta(y \mid x) = \prod_{t} \pi_\theta(y_t \mid x, y_{<t})$	The language model as a policy over tokens
$\pi_\text{ref}$	Frozen reference model (usually the SFT model)
$r(x, y)$	Reward for the whole response; $r_\phi$ if it is a learned reward model with parameters $\phi$
$\beta$	KL penalty coefficient (strength of the leash to $\pi_\text{ref}$ )
$\mathrm{KL}(\pi_\theta \,\Vert\, \pi_\text{ref})$	KL divergence from the reference model
$(x, y_w, y_l)$	A preference pair: $y_w$ preferred (“won”) over $y_l$ (“lost”)
$\sigma(z) = 1/(1+e^{-z})$	Logistic sigmoid
$G$	Group size: number of responses sampled per prompt (GRPO)
$y^{(i)}$ , $i = 1..G$	The $i$ -th sampled response for a prompt; $y^{(i)}_t$ is its $t$ -th token
$r^{(i)}$	Reward of response $y^{(i)}$
$\hat{A}^{(i)}$	Advantage of response $y^{(i)}$ , e.g. $(r^{(i)} - \text{mean}) / \text{std}$ in GRPO