Back to interview questions
By
Team Sherlock.sh
Reinforcement learning interviews reveal whether a candidate can translate a sequential decision problem into a defensible objective, learning method, and evaluation plan. Recruiters need to distinguish genuine understanding of feedback, uncertainty, and delayed consequences from familiarity with algorithm names. A high simulated reward is not enough when the policy exploits a flawed reward or fails outside its training environment.
This guide to 100+ Reinforcement Learning Interview Questions and Answers contains 120 questions across beginner, intermediate, and experienced levels. It covers decision-process foundations, value-based and policy-based methods, deep reinforcement learning, model-based and offline learning, evaluation, safety, and deployment. Use the examples to examine reasoning and implementation skill without assuming that any algorithm is best for every task.
How can recruiters identify potential AI cheating, deepfakes, impersonation, and unauthorized assistance during reinforcement learning interviews?
Ask candidates to calculate a Bellman target, explain why a time-limit cutoff changes bootstrapping, or diagnose a policy that exploits its reward. Define permitted tools in advance, then ask follow-up questions that connect the candidate’s code to the assumptions behind the update.
Sherlock AI helps recruiters detect potential AI-assisted cheating, deepfakes, candidate impersonation, and unauthorized real-time assistance during reinforcement learning interviews.
Combine interview-integrity checks with a small decision-process exercise and discussion of evaluation evidence. Interpret suspicious signals in context rather than treating one signal as proof of misconduct.
Reinforcement Learning Interview Questions
Select questions for the role and state the environment, objective, data access, action constraints, and permitted tools. Distinguish algorithm recall from the ability to reason about data collection, estimation error, and deployment consequences.
Reinforcement Learning Interview Questions for Beginners
Establish the language of sequential decision-making: states, observations, actions, rewards, policies, returns, value functions, and basic learning updates.
What is reinforcement learning, and how does it differ from supervised learning?
What are the agent, environment, state, action, and reward?
What is a Markov decision process?
What does the Markov property mean in practice?
How does a partially observable problem differ from an MDP?
What is a policy, and can it be stochastic?
What is a return, and how is it calculated?
Why do reinforcement learning algorithms use a discount factor?
How do episodic and continuing tasks differ?
How do termination and truncation differ?
What are state-value and action-value functions?
What is the Bellman expectation equation?
What is the Bellman optimality equation?
What is the difference between prediction and control?
How do model-based and model-free reinforcement learning differ?
What does dynamic programming require in reinforcement learning?
How does iterative policy evaluation work?
How does policy improvement lead to policy iteration?
How does value iteration differ from policy iteration?
How does Monte Carlo prediction estimate a value function?
What is the difference between first-visit and every-visit Monte Carlo?
What is temporal-difference learning?
What does bootstrapping mean in reinforcement learning?
How do Monte Carlo and temporal-difference methods compare?
How does the SARSA update work?
How does the Q-learning update work?
Why are SARSA and Q-learning classified differently as on-policy and off-policy?
What is the exploration–exploitation trade-off?
How does an epsilon-greedy policy choose actions?
What is a multi-armed bandit, and how does it differ from an MDP?
What is optimistic initialization, and when can it help exploration?
What roles do the learning rate and discount factor play?
What is the difference between a reward and a value?
Why are sparse rewards difficult to learn from?
What is reward shaping, and what can go wrong?
What makes a state or action representation useful?
How should a basic Gymnasium interaction loop handle episode boundaries?
How would you evaluate a simple reinforcement learning agent?
What common implementation bugs would you check in tabular reinforcement learning?
When is reinforcement learning an inappropriate choice?
Reinforcement Learning Interview Questions for Intermediate
Probe implementation knowledge of deep value learning, policy gradients, actor-critic methods, advantage estimation, replay, and continuous control.
How do n-step returns connect Monte Carlo and temporal-difference learning?
What are eligibility traces and TD(lambda)?
What changes when a value function uses function approximation?
What is the deadly triad in reinforcement learning?
How does DQN turn Q-learning into a neural-network algorithm?
Why does experience replay help, and when can it hurt?
Why does DQN use a separate target network?
How does Double DQN reduce maximization bias?
What does a dueling Q-network represent?
How does prioritized experience replay change sampling and correction?
What does distributional reinforcement learning predict?
What is the policy gradient theorem useful for?
How does REINFORCE estimate a policy gradient?
Why can an action-independent baseline reduce variance without bias?
What does an actor–critic algorithm do?
How do advantage estimates guide a policy update?
How does generalized advantage estimation work?
How should GAE handle termination, truncation, and rollout boundaries?
How do A2C and A3C differ?
What does the PPO clipped objective optimize?
Does PPO clipping guarantee a trust region or monotonic improvement?
What should a practical PPO update keep fixed and monitor?
How does TRPO differ from PPO?
How does a deterministic policy gradient work?
What are the main components and failure modes of DDPG?
How does TD3 address errors in deterministic actor–critic learning?
What objective does soft actor–critic optimize?
How are the SAC critic and actor losses constructed?
How does a squashed Gaussian policy handle bounded continuous actions?
What does entropy regularization change about learning?
How does importance sampling support off-policy learning?
Why can trajectory importance sampling have high variance?
What is the difference between ordinary and weighted importance sampling?
Why are off-policy multi-step corrections needed?
How should a replay buffer represent terminal and truncated transitions?
How can recurrent policies learn from partially observed trajectories?
How do observation and reward normalization affect an RL implementation?
Which numerical and gradient checks catch common actor–critic bugs?
How do action masks interact with policy probabilities and training?
How would you choose between DQN, PPO, and SAC for a bounded experiment?
Reinforcement Learning Interview Questions for Experienced
Assess judgment about model-based planning, offline data, trustworthy evaluation, reward design, safety, and deployment under uncertainty.
What does model-based reinforcement learning add beyond a model-free policy?
How does Dyna combine learning from experience with planning?
How would you implement model-predictive control with a learned dynamics model?
Why can an accurate one-step model still produce a poor policy?
How do world models handle partially observed environments?
What makes offline reinforcement learning different from ordinary off-policy learning?
How do you assess dataset coverage before attempting offline RL?
How does Conservative Q-Learning reduce unsupported value optimism?
How does Implicit Q-Learning avoid querying unseen actions during training?
When is behavior cloning a better starting point than offline RL?
How does trajectory importance sampling estimate an unseen policy's return?
Why do importance weights become unstable, and what can you do about it?
What makes an off-policy value estimator doubly robust?
How does fitted Q evaluation work, and where can it fail?
How would you decide whether offline evaluation is sufficient for deployment?
How do you design a statistically credible RL benchmark comparison?
How do you prevent evaluation leakage in an RL project?
What must be recorded to make an RL result reproducible?
How do you distinguish training instability from an environment integration bug?
How do you address the simulation-to-reality gap?
How do constrained MDPs represent safety requirements?
Why is a reward penalty not the same as a hard safety guarantee?
How can potential-based reward shaping preserve an optimal policy?
How would you detect and respond to reward hacking?
How do you tackle long-horizon credit assignment?
What are options in hierarchical reinforcement learning?
Why does multi-agent learning create nonstationarity?
How does centralized training with decentralized execution work?
How would you formulate a recommendation problem without optimizing only clicks?
How would you design a bounded RL controller for a physical process?
How do you choose between RL, a contextual bandit, and conventional optimization?
How do you detect distribution shift after an RL policy is deployed?
What should a safe rollout and rollback plan contain?
Why must action permissions be enforced outside the learned policy?
How does reinforcement learning from human feedback differ from ordinary RL?
What is the standard preference-based reward-model training objective?
How does PPO-based RLHF balance reward improvement and reference-policy drift?
How does Direct Preference Optimization differ from PPO-based RLHF?
How can preference optimization overfit or exploit a learned reward?
How would you review an RL system before approving a production experiment?
Reinforcement Learning Interview Questions and Answers
The answers match the question bank and lead with direct explanations candidates can adapt. Equations use explicitly stated assumptions; small Python examples and pseudocode illustrate update rules rather than complete training systems. Validate library-specific behavior in the installed version before using an example in a real project.
Reinforcement Learning Interview Questions and Answers for Beginners
These answers connect foundational definitions to small worked examples and clarify where episodic and discounted assumptions matter.
1. What is reinforcement learning, and how does it differ from supervised learning?
Reinforcement learning learns how to act through interaction, using rewards to evaluate consequences rather than receiving a correct action label for every input. The objective is usually expected cumulative reward, so an action can be useful even when its immediate reward is poor. Supervised learning instead fits predictions to supplied targets, although an RL system can contain supervised components such as a learned transition model.
For example, a warehouse robot may take a longer route now to avoid congestion that delays future deliveries. The training signal must connect that delayed outcome to earlier decisions. Actions also influence which states and data the robot encounters, unlike the usual fixed supervised dataset assumption.
I would first ask whether decisions materially affect future opportunities. If each decision is independent and labeled outcomes are available, supervised learning or a contextual bandit may be simpler. RL is a problem framework, not a guarantee that trial-and-error deployment is safe or efficient.
2. What are the agent, environment, state, action, and reward?
The agent is the decision-making component, and the environment is everything it interacts with outside that component. At time t, the agent uses available information to choose action A_t; the environment then produces reward R_(t+1) and the next state or observation. A state summarizes the information needed to predict future dynamics, whereas an observation is what the agent actually receives.
For a thermostat, the action could be a heating level, the observation could include measured temperature, and reward could penalize energy use and discomfort. The true physical state may additionally include hidden wall temperatures or occupancy. Calling a sensor reading a state does not make it Markov.
The boundary is a modeling choice: a low-level controller may belong to the agent in one design and the environment in another. I define timing, units, action constraints, and reward computation explicitly so that the learning problem is reproducible rather than relying on informal labels.
3. What is a Markov decision process?
A Markov decision process specifies states, available actions, transition and reward dynamics, and an objective such as discounted return. A useful joint dynamics notation is p(s′, r | s, a): after action a in state s, it gives the probability of next state s′ and reward r. A policy then determines how actions are selected.
For example, a gridworld action may move right with probability 0.8 and slip upward with probability 0.2. Its reward can depend on the resulting transition, such as entering a goal or a hazard. Deterministic movement is just a special case of this probabilistic model.
An MDP does not imply that the agent knows the dynamics, that the environment is deterministic, or that rewards are positive. The essential assumption is that the specified current state and action are sufficient for the next-step distribution. For finite-horizon problems, remaining time may need to be part of the state or the value function's time index.
4. What does the Markov property mean in practice?
The Markov property means that the current state contains all past information relevant to predicting the next state and reward, once the current action is known. Formally, conditioning on the full history should not change the next-step distribution beyond conditioning on S_t and A_t. It does not mean that states are independent across time.
Position alone is usually not a Markov state for a moving vehicle: two vehicles at the same position but with different velocities have different likely next positions. Including velocity, and any other relevant physical variables, makes the representation more adequate. Likewise, an inventory state may need outstanding orders as well as stock on hand.
I treat Markov sufficiency as a modeling assumption to inspect, not a property guaranteed by putting features in a vector. History windows, recurrent memory, or belief estimates can help when observations omit relevant information, but a fixed history window is not universally sufficient.
5. How does a partially observable problem differ from an MDP?
A partially observable Markov decision process has an underlying Markov state that the agent cannot directly observe. The agent receives observations generated from that state, so two identical observations can require different actions depending on history. A memoryless observation-to-action policy can therefore lose information needed for good decisions.
For example, a robot may see the same corridor view while facing different hidden destinations. Maintaining a history or an estimated belief distribution over possible states can disambiguate the situation. With known dynamics and observation probabilities, Bayesian filtering updates that belief after each action and observation; the belief is a sufficient information state under the model.
A recurrent network is an alternative learned memory mechanism, not proof that partial observability has been solved. I would check which hidden variables matter, what observations reveal them, and whether extra sensing is possible. Exploration can include information-gathering actions whose immediate reward is small but whose observations improve later decisions.
6. What is a policy, and can it be stochastic?
A policy specifies how an agent chooses actions from its available information. In an MDP, a stationary stochastic policy can be written π(a | s), a probability distribution over actions in state s. A deterministic policy selects one action, often written a = π(s). Stationary means the rule does not explicitly change with time; finite-horizon policies may instead depend on the remaining horizon.
For example, at an intersection a policy might turn left with probability 0.7 and right with probability 0.3. A value-based agent can induce a policy by choosing the action with the highest estimated Q-value, whereas a policy-based method directly parameterizes action probabilities or action outputs.
Stochastic policies are useful for exploration and can be important under partial observability or adversarial interaction. They are not inherently less learned or less optimal. In a finite discounted fully observed MDP, a deterministic stationary optimal policy exists, but that result should not be generalized to every constrained or partially observed problem.
7. What is a return, and how is it calculated?
A return is the cumulative future reward associated with a time step, under a specified horizon and discounting rule. For an episode ending at T, G_t = Σ from k = 0 to T−t−1 of γ^k R_(t+k+1). The reward immediately following the current action is not discounted; later rewards receive increasing powers of γ.
For rewards 2, 0, and 5 with γ = 0.9, the initial return is 2 + 0.9 × 0 + 0.81 × 5 = 6.05. A backward calculation is convenient:
Python example
This example is a complete terminal trajectory. For a truncated trajectory representing an unfinished continuing task, initializing the tail to zero changes the target; a bootstrapped continuation estimate may be needed instead. A sampled return is one random outcome, while a value function is its conditional expectation.
8. Why do reinforcement learning algorithms use a discount factor?
The discount factor determines how much future rewards contribute to the objective. With 0 ≤ γ < 1, a reward k steps ahead is multiplied by γ^k. Discounting can express a preference for earlier outcomes and makes infinite-horizon returns bounded when rewards are bounded. It also supports contraction-based results for finite discounted MDPs.
For a constant reward of 1 forever, the discounted return is 1/(1−γ): it is 10 when γ = 0.9 and 100 when γ = 0.99. This illustrates why values and effective planning horizons change substantially with γ. The rough scale 1/(1−γ) is not a hard cutoff after which rewards disappear.
Setting γ to zero makes the objective myopic. Setting it to one can be appropriate for episodic tasks with well-defined finite expected returns, but it does not automatically work for arbitrary continuing tasks. I choose γ to match the task objective, not merely as a knob to make a training curve look better.
9. How do episodic and continuing tasks differ?
An episodic task has meaningful terminal events that divide interaction into episodes, such as completing a delivery or losing a game. A continuing task has no natural terminal boundary, such as ongoing resource allocation. Both can use discounted returns; episodic tasks can also use undiscounted total rewards when their returns are well-defined.
A terminal state is conventionally assigned zero future value because no rewards remain in that episode. This does not mean the transition into it has zero reward: a final success bonus is still included. A reset begins a new episode rather than becoming a normal continuation transition for the previous episode's value target.
A continuing simulation can be collected in finite chunks for practical reasons. Those collection boundaries do not turn the underlying task into an episodic one. I distinguish the task horizon, rollout length, and evaluation duration explicitly because confusing them can change both learning targets and reported performance. Average-reward formulations are another option for some continuing objectives.
10. How do termination and truncation differ?
Termination means the task reaches an actual terminal state; truncation means data collection stops for an external reason while the modeled task could continue. A genuine finite-horizon deadline can be termination, with remaining time represented appropriately. A wrapper's arbitrary collection limit for a continuing task is truncation.
For a one-step value target, termination removes the continuation value, but truncation alone does not:
Python example
Either signal requires starting a new episode in an ordinary Gymnasium loop. The bootstrap must use the actual final observation, not an automatically reset observation. An invalid final state or different task definition needs explicit handling rather than blindly assigning a continuation estimate.
11. What are state-value and action-value functions?
A state-value function gives the expected return from a state when following a particular policy: v_π(s) = E_π[G_t | S_t = s]. An action-value function additionally conditions on taking a specific first action: q_π(s, a) = E_π[G_t | S_t = s, A_t = a], followed by that policy thereafter. Values describe expected future reward, not action probabilities.
They are related by v_π(s) = Σ_a π(a | s) q_π(s, a). If a policy chooses two actions equally and their Q-values are 4 and 10, the state's value is 7. Choosing the higher-value action instead would change the policy and potentially its later state values.
I distinguish the true value functions from estimates such as V and Q stored in a table or neural network. A high learned Q-value may reflect estimation error rather than a genuinely good action. Both values depend on the reward definition, discount factor, dynamics, and policy, so numbers are not comparable across arbitrarily different tasks.
12. What is the Bellman expectation equation?
The Bellman expectation equation expresses a policy's value as expected immediate reward plus discounted value of the next state. For a finite MDP, v_π(s) = Σ_a π(a | s) Σ_(s′,r) p(s′, r | s, a)[r + γv_π(s′)]. It averages over both the policy's action choices and the environment's random outcomes.
Suppose the policy always takes an action that yields reward 2 and reaches a state worth 6. With γ = 0.5, the current state's value is 2 + 0.5 × 6 = 5. If there are several possible next states, their continuation values must be weighted by their probabilities rather than choosing the most favorable outcome.
The equation is an identity for the true value function, not automatically a learning algorithm. Policy evaluation uses it to construct updates or solve a linear system when a model is available. Sample-based temporal-difference learning uses observed transitions instead of computing the entire expectation.
13. What is the Bellman optimality equation?
The Bellman optimality equation replaces the policy's average over actions with a maximum, reflecting the best achievable expected return. In a finite discounted MDP, v_(s) = max_a Σ_(s′,r) p(s′, r | s, a)[r + γv_(s′)]. Similarly, q_(s, a) averages r + γ max_(a′) q_(s′, a′) over the next transition.
For two deterministic actions with immediate-reward/next-value pairs (1, 8) and (4, 2), γ = 0.5 gives action scores 5 and 5. Both are greedy choices under those continuation values. The maximization selects an action before observing the random outcome; it does not grant control over random transitions.
A policy that is greedy with respect to the correct optimal value function is optimal under the stated MDP assumptions. Being greedy with respect to an inaccurate learned estimate is not the same guarantee. Approximation error, missing state information, and insufficient exploration can all prevent the estimate from satisfying the optimality equation.
14. What is the difference between prediction and control?
Prediction estimates the consequences of a specified policy, whereas control seeks a better policy. Evaluating the expected return of a fixed warehouse routing rule is prediction. Adjusting the route choices to improve expected return is control. Both problems may use the same value-function representation and experience stream, but their targets differ.
For prediction, the Bellman equation averages future actions according to the policy being evaluated. For control, methods repeatedly improve action choices, for example by making them greedy or epsilon-greedy with respect to an evolving Q-function. This interaction between evaluation and improvement is often called generalized policy iteration.
I make the target policy explicit even when another behavior policy collects the data. Otherwise, a reported value can be ambiguous: it might estimate the exploratory training policy, the greedy deployment policy, or neither if the update is incorrect. Prediction quality also does not directly establish control quality; small average value errors can still reverse the ranking of actions in important states.
15. How do model-based and model-free reinforcement learning differ?
Model-based methods use a model of environmental transitions and rewards to reason about possible consequences. The model may be supplied or learned from experience. Model-free methods learn values or policies without using an explicit transition model for planning. Both can learn from interaction, and both can use function approximation.
For example, a planner can simulate candidate action sequences through a learned dynamics model before selecting a move. A tabular Q-learning agent instead updates action values directly from observed transitions. A hybrid can update a model from real experience and also generate simulated transitions for additional learning.
Model-based is not synonymous with perfect knowledge, and model-free is not synonymous with neural networks. The useful trade-off is whether planning with a model saves real interactions enough to justify model fitting and computation. Model errors can compound over imagined trajectories, while model-free approaches can require substantial data. I would compare methods under the actual interaction budget and task structure rather than declaring one category universally superior.
16. What does dynamic programming require in reinforcement learning?
Classical dynamic programming requires a usable model of transitions and rewards so that Bellman backups can average over possible outcomes. In its tabular form, it also assumes a manageable state and action space. Policy evaluation, policy iteration, and value iteration are standard dynamic-programming approaches for finite MDPs.
For a candidate action, a backup computes Σ_(s′,r) p(s′, r | s, a)[r + γV(s′)]. This differs from a sample update that uses only one observed reward and successor. A known simulator can generate samples without exposing the full probabilities, so having simulator access is not automatically equivalent to having an enumerable model for exact backups.
The practical limitation is the cost of representing and sweeping all relevant states and actions. A robot's continuous state space cannot usually be handled by a literal table. Approximation, discretization, sampling, or other planning methods may be necessary, and their errors require separate analysis. Discounted finite-MDP convergence results do not automatically transfer unchanged to every approximation scheme.
17. How does iterative policy evaluation work?
Iterative policy evaluation repeatedly applies the Bellman expectation backup for a fixed policy until the value estimates are sufficiently stable. In synchronous evaluation, every state in a sweep uses values from the previous sweep. In-place variants use newly updated values as soon as they are available; their update ordering should be stated.
For a one-state continuing problem with reward 1, a fixed self-loop, and γ = 0.9, the exact value is 10. Starting from zero:
Python example
For finite MDPs with γ below one, the policy Bellman operator is a contraction and repeated exact backups converge to the policy's value. A small numerical stopping tolerance approximates that fixed point; it is not evidence that the environment model is correct. With γ equal to one, additional episodic/proper-policy conditions are needed rather than assuming the same contraction argument.
18. How does policy improvement lead to policy iteration?
Policy improvement changes a policy to favor actions with higher expected return under its current value function. After evaluating π, compute q_π(s, a) using one-step lookahead and select an action maximizing it. The policy improvement theorem establishes nondecreasing value under the standard exact finite-MDP setting.
Policy iteration alternates evaluation and improvement until the policy is stable. For example, if the evaluated policy's chosen action has lookahead value 3 while an alternative has value 5, switching to the alternative is a justified improvement step. The comparison uses the old policy's continuation values; evaluating the new policy then accounts for all its changed future decisions.
Tie handling matters operationally: changing between equally good actions can make a naive implementation appear unstable. A consistent tie rule avoids this. Exact evaluation can be expensive, so practical variants may perform limited evaluation sweeps. With noisy estimates or function approximation, a greedy change can degrade real performance; the exact theorem is not a blanket guarantee for approximate implementations.
19. How does value iteration differ from policy iteration?
Value iteration combines evaluation and improvement in each backup by directly applying the Bellman optimality operator. Instead of fully evaluating a policy before changing it, it updates V(s) toward the best one-step lookahead value. After convergence, a greedy policy is extracted from those values and the transition model.
For a finite discounted MDP, the update is V_new(s) = max_a Σ_(s′,r) p(s′, r | s, a)[r + γV_old(s′)]. Synchronous sweeps use a separate old-value table; in-place implementations have different scheduling but can also converge under suitable conditions. Terminal states need the chosen zero-continuation convention.
Policy iteration can require fewer policy changes but more work evaluating each policy. Value iteration performs cheaper partial evaluation steps but may need many sweeps, especially with a discount factor near one. Neither is universally faster. I would compare total backup cost and the required policy-quality tolerance, not simply the number of outer iterations, and verify that the model and finite-state assumptions actually hold.
20. How does Monte Carlo prediction estimate a value function?
Monte Carlo prediction estimates a policy's value by averaging returns observed after visits to a state. It does not need transition probabilities and does not bootstrap from a value estimate. In the basic episodic setting, it waits for the episode to end so that the complete sampled return is available.
If returns from a state are 3, 7, and 5, their sample mean is 5. An incremental mean avoids storing all returns:
Python example
The estimate concerns the policy that generated those returns unless appropriate off-policy correction is used. Its reliability also depends on enough relevant visits and well-defined return moments. Long, noisy episodes can produce high-variance targets. A fixed step size can track changing values, but unlike a diminishing sample-mean step size it does not simply compute the average of all historical returns.
21. What is the difference between first-visit and every-visit Monte Carlo?
First-visit Monte Carlo averages the return following the first visit to a state in each episode. Every-visit Monte Carlo uses the return following every occurrence of that state. Both target the policy's state value under their usual episodic sampling assumptions, but the samples and their statistical dependencies differ.
Suppose an episode visits state A twice, with returns 8 and 3 after those visits. First-visit prediction contributes only 8 for A from that episode; every-visit prediction contributes both 8 and 3. A second visit's return includes a shorter suffix of the same trajectory, so those observations are not independent episode samples.
A common implementation error is to scan backward while keeping a seen-state set and accidentally retain the last chronological visit. To implement first-visit updates, explicitly identify first occurrence indices or check whether the state appeared earlier in the trajectory. The same distinction applies to state-action pairs in Monte Carlo action-value estimation. Neither variant removes the need for sufficient coverage or makes a truncated trajectory a complete return.
22. What is temporal-difference learning?
Temporal-difference learning updates a value estimate using an observed reward and an estimated continuation value. TD(0) uses δ_t = R_(t+1) + γV(S_(t+1)) − V(S_t), followed by V(S_t) ← V(S_t) + αδ_t. At true termination, the continuation term is zero.
If the current estimate is 4, the reward is 2, the next estimate is 6, γ = 0.9, and α = 0.1, the target is 7.4, the error is 3.4, and the updated estimate is 4.34:
Python example
TD can learn before an episode finishes and does not require a transition model. Its targets depend on imperfect current estimates, so stability and convergence depend on the representation, sampling policy, learning rates, and other assumptions rather than the update formula alone.
23. What does bootstrapping mean in reinforcement learning?
Bootstrapping means using an existing estimate as part of the target for another estimate. In TD(0), reward plus γV(next state) is a bootstrapped target because the continuation value is not a fully observed future return. Dynamic-programming backups also bootstrap even though they average over a model rather than sampling one transition.
For example, a robot can improve its estimate for the current location after one move by combining the observed travel cost with its estimated value of the new location. It need not wait for the entire delivery to finish. If that next-state estimate is wrong, however, some of its error can propagate backward.
Monte Carlo prediction with complete episode returns does not bootstrap. This distinction is separate from sampling: TD samples and bootstraps, Monte Carlo samples without bootstrapping, and exact dynamic programming bootstraps without sampling transitions. Bootstrapping is not inherently biased in every final estimate, but individual targets can be inaccurate while their continuation estimates are still learning.
24. How do Monte Carlo and temporal-difference methods compare?
Monte Carlo methods use complete sampled returns, while temporal-difference methods combine sampled rewards with estimated future values. Basic Monte Carlo prediction waits until an episode ends. TD can update after each transition, making it naturally useful for continuing tasks and online learning.
Monte Carlo targets can have substantial variance because they incorporate the randomness of many future rewards and actions. One-step TD often reduces that source of variance but introduces dependence on the current continuation estimate. This is a practical bias–variance trade-off, not a universal statement that one method always learns faster or reaches a better policy.
For a short deterministic episodic task with reliable completed trajectories, Monte Carlo may be straightforward. For long tasks with delayed completion, TD can propagate useful information earlier. Multi-step targets interpolate between the extremes. I would compare data efficiency and final policy quality under a shared evaluation protocol, and I would not treat a collection cutoff as an actual terminal event merely to make a Monte Carlo implementation convenient.
25. How does the SARSA update work?
SARSA is a temporal-difference control method whose update uses the next action actually selected by the behavior policy. Its name refers to the tuple state, action, reward, next state, next action. The nonterminal update is Q(s, a) ← Q(s, a) + α[r + γQ(s′, a′) − Q(s, a)].
Suppose Q(s, a) = 4, r = 1, Q(s′, a′) = 2, γ = 0.9, and α = 0.5. The target is 2.8 and the new value is 3.4. If the exploratory policy occasionally chooses risky future actions, SARSA's target reflects those choices rather than assuming perfect greedy behavior afterward.
At true termination, there is no next action to bootstrap from, and the target is just r. At an external truncation, a continuation target can still be appropriate for the underlying task. In tabular control, optimal convergence requires exploration and learning-rate conditions; fixed epsilon generally evaluates and improves an exploratory policy rather than simply eliminating exploration risk.
26. How does the Q-learning update work?
Q-learning updates an action value toward the observed reward plus the largest estimated next-state action value. The target is r + γ max_(a′) Q(s′, a′) for a nonterminal transition, regardless of which action the behavior policy actually chooses next. This makes tabular Q-learning an off-policy control algorithm.
Python example
The maximum ranges over valid next-state actions, and terminal transitions must not accidentally use a reset state's Q-values. For finite discounted MDPs, convergence results require sufficient state-action visitation and suitable diminishing step sizes, alongside the usual boundedness conditions. They do not automatically apply to arbitrary neural-network Q-learning. Greedy targets also do not remove the need for exploration in the data-collection policy.
27. Why are SARSA and Q-learning classified differently as on-policy and off-policy?
SARSA is on-policy because its target uses the next action selected under the policy whose behavior it is learning. Q-learning is off-policy because its target uses a greedy next action even when an exploratory behavior policy generated the transition. The distinction is between the target policy and the data-collection policy, not between online and offline data storage.
Imagine an agent walking near a cliff. An epsilon-greedy SARSA agent accounts for the risk that future exploration will choose a dangerous action. Q-learning estimates the greedy continuation's value, so its preferred route can differ even when both agents collect data with the same epsilon-greedy rule. This is a useful illustration, not a universal performance ranking.
Off-policy does not mean any dataset is sufficient. If important actions are never tried, their values may remain unsupported. Nor does every off-policy method use an explicit importance-sampling ratio: the one-step tabular Q-learning target has a special structure that supports off-policy control without the trajectory reweighting used in some other estimators.
28. What is the exploration–exploitation trade-off?
Exploration gathers information about uncertain actions, while exploitation selects actions that currently appear most rewarding. The trade-off exists because choosing an action affects both the immediate outcome and what the agent can learn. Always exploiting an inaccurate early estimate can lock an agent into a poor policy.
For example, a delivery planner may repeatedly select a familiar route worth an estimated 5 while ignoring an untested route that could be worth 8 or 2. Trying the second route can improve later decisions, but it incurs opportunity cost and potentially operational risk. In an MDP, exploration also determines which future states become reachable, so independent random action choices may not discover distant rewards efficiently.
I separate useful exploration from unrestricted real-world experimentation. Constraints, safe simulators, demonstrations, or approved exploration policies may be necessary. The right exploration strategy depends on uncertainty, horizon, nonstationarity, and the cost of mistakes. A fixed amount of random behavior is a mechanism, not a guarantee of meaningful coverage or safe learning.
29. How does an epsilon-greedy policy choose actions?
An epsilon-greedy policy exploits a highest-valued action with probability 1−ε and explores by sampling from the action set with probability ε. Under uniform exploration across k actions and one uniquely greedy action, that greedy action has total probability 1−ε+ε/k, because exploration can select it too.
With ε = 0.2 and four actions, the greedy action receives probability 0.85 and each other action 0.05:
Python example
Tie-breaking should be explicit, often randomized across maximizers. Decaying epsilon too quickly can stop exploration before useful actions are discovered. Conversely, a positive fixed epsilon retains ongoing exploratory mistakes. The probability example assumes all four actions are valid; action masks change the exploration distribution and must be applied consistently to selection and value backups.
30. What is a multi-armed bandit, and how does it differ from an MDP?
A multi-armed bandit repeatedly chooses among actions and observes rewards, without modeling action-dependent transitions into future decision states. The central problem is learning which actions pay off while balancing exploration and exploitation. An MDP adds sequential state transitions, so an action can change the future situations and rewards available.
Choosing among fixed advertising variants for independent impressions is a simple bandit abstraction. Deciding how to manage a customer relationship over many interactions may require an MDP because current actions can affect later engagement. A contextual bandit additionally conditions each action on current context, but it still does not optimize action-driven long-term state changes in the MDP sense.
I would choose the least complex model that captures the decision's causal structure. Using full RL where immediate outcomes suffice adds estimation and evaluation difficulty. Conversely, treating a sequential problem as a bandit can optimize short-term rewards while missing downstream harm. The modeling distinction concerns the decision process, not whether the implementation uses a neural network.
31. What is optimistic initialization, and when can it help exploration?
Optimistic initialization sets initial action-value estimates above plausible typical returns so that untried actions remain attractive. As disappointing outcomes lower the estimates of tried actions, a greedy agent is encouraged to try alternatives. It is a way to influence exploration through estimates rather than adding random actions directly.
In a stationary bandit whose rewards lie between zero and one, initializing every action value to 2 can cause initial experimentation. The exact behavior depends on tie-breaking and step sizes. In a discounted MDP, an optimistic value must be considered relative to return scale, not just one-step reward scale; bounded reward 1 with γ = 0.99 allows returns up to 100.
This technique mainly creates an initial incentive. Once optimism disappears, a purely greedy agent may fail to explore changes in a nonstationary environment. Function approximation can also generalize estimates in unexpected ways. I would not treat optimistic values as calibrated uncertainty or as a guarantee that every important state-action pair will be visited.
32. What roles do the learning rate and discount factor play?
The learning rate α controls how strongly a new target changes the current estimate, while the discount factor γ defines how future rewards enter the objective. In Q ← Q + α(target−Q), a larger α reacts faster but can amplify noise. Changing γ changes the return being optimized, not just the speed of fitting it.
For a current estimate 10 and a target 6, α = 0.1 gives 9.6, while α = 0.5 gives 8. Both updates use the same target. Increasing γ, by contrast, increases the weight of estimated continuation rewards and can change which action is optimal.
Diminishing learning rates support some tabular convergence results when each relevant state-action pair receives sufficiently many updates and the step-size sums meet the required conditions. Constant learning rates can help track nonstationarity but generally retain fluctuations. I would tune α against stability and sample efficiency while selecting γ based on task horizon and objectives, rather than interpreting both parameters as interchangeable optimization settings.
33. What is the difference between a reward and a value?
Reward is a signal emitted by the environment for a transition; value is an expectation of accumulated future rewards under a policy. Reward says how the task scores an immediate outcome. Value predicts how promising a state or action is over the chosen horizon. A value estimate is learned or computed, not usually supplied as the environment's feedback.
Paying an energy cost of −2 to recharge can be locally negative yet have high action value if it enables later deliveries worth 20. Conversely, collecting a small immediate bonus can have poor value if it traps the agent in a bad state. This is why greedily maximizing one-step reward is generally different from maximizing return.
The reward function is still the formal optimization target, so a high return only means success according to that definition. It need not imply that the system met every human requirement. I keep business outcomes, constraints, and diagnostic metrics alongside reward, rather than treating the learned value as a complete measure of real-world desirability.
34. Why are sparse rewards difficult to learn from?
Sparse rewards provide little direct feedback on most transitions, so the agent may rarely encounter informative success or failure signals. This creates both an exploration problem—finding a rewarding trajectory—and a credit-assignment problem—identifying which earlier actions contributed. A long horizon can make random exploration particularly inefficient.
For example, a robot rewarded only after assembling a complete object receives identical zero rewards for many useful and useless partial attempts. Demonstrations, curricula, better exploration, or carefully designed intermediate feedback can make learning more feasible. None automatically solves the problem: a curriculum can teach behavior that transfers poorly, and intermediate rewards can redirect the objective.
I would inspect whether successful trajectories are reachable with the proposed exploration budget before choosing a more elaborate optimizer. If almost no training episodes ever reach the goal, merely increasing the learning rate cannot manufacture missing evidence. Evaluation should still measure the original task success, not only a denser training reward that may be easier to optimize.
35. What is reward shaping, and what can go wrong?
Reward shaping adds feedback intended to make useful behavior easier to learn, but arbitrary extra rewards can change the optimal policy. Rewarding a robot whenever it approaches an object may teach repeated approach-and-retreat cycles instead of task completion. The agent is optimizing the implemented signal, not the designer's unexpressed intention.
A standard policy-preserving construction under appropriate MDP conditions adds F(s, a, s′) = γΦ(s′) − Φ(s), where Φ is a state potential and γ matches the original discount. The discounted shaping terms telescope, leaving a policy-independent shift when boundary conditions are handled correctly. For episodic use, terminal potentials must be chosen appropriately, commonly zero.
I would validate shaping against the original objective and explicitly test loops, premature termination, and reward-collection shortcuts. Potential-based shaping does not guarantee fast learning, safe exploration, or correctness under arbitrary approximations. It gives a specific policy-invariance result under stated assumptions, which is much narrower than saying that any helpful-looking bonus is harmless.
36. What makes a state or action representation useful?
A useful state representation retains the information needed for decisions and prediction without introducing avoidable complexity or leakage. A useful action representation corresponds to decisions the agent can actually execute, with clear units, timing, and validity constraints. Both choices determine the effective learning problem before an algorithm is selected.
For inventory control, current stock alone may be inadequate; pending deliveries and remaining lead times can affect the next state. For robot control, a discrete direction choice and a continuous torque vector imply very different action spaces. Excessive discretization can remove important precision, while an unnecessarily large action set can make exploration difficult.
I would check that features are available at decision time and do not expose future outcomes or hidden simulator information unavailable at deployment. Normalization statistics should be fitted without evaluation leakage. If the task has a real deadline, remaining time may be essential. A compact representation is helpful only if it preserves the distinctions that change good actions; dimensionality reduction is not automatically decision-preserving.
37. How should a basic Gymnasium interaction loop handle episode boundaries?
A basic loop resets the environment, chooses an action from the current observation, then processes the next observation, reward, and two boundary flags. It resets after either termination or truncation, but stores those flags separately so learning code can choose the correct continuation target.
Python example
Here policy, store, and step_budget are application-supplied placeholders, not a runnable training implementation. This loop assumes ordinary single-environment behavior rather than an autoresetting wrapper. With autoreset or vectorized environments, verify how the true final observation is returned before constructing targets. Seeding environment resets does not automatically seed every policy or library random-number generator.
38. How would you evaluate a simple reinforcement learning agent?
I would evaluate a frozen policy on separate episodes drawn from the intended deployment conditions and report distributions, not just the best return. Useful measurements include return, success rate, episode length, constraint violations, and resource costs. The evaluation policy must be specified: greedy, stochastic, or exploratory behavior can produce different results.
For example, ten successful episodes out of ten do not establish a near-perfect success probability, especially if all episodes share one easy starting condition. I would vary relevant initial conditions and random seeds, preserve a held-out test protocol, and compare against simple baselines under a comparable interaction budget. Training return is useful for diagnosis but not a substitute for this evaluation.
Repeatedly choosing hyperparameters on the test set leaks information into the final result. A validation split or separate environment configurations help preserve a meaningful final assessment. Small local calculations can validate update arithmetic, but they do not establish learning convergence, robust policy performance, or safe deployment. Those claims require appropriately designed experiments.
39. What common implementation bugs would you check in tabular reinforcement learning?
I would first check transition alignment, terminal handling, action indexing, and whether the update uses the intended old or new values. The stored reward must belong to the stored action, and a target must never bootstrap from the next episode's reset state. Confusing observation indices with action indices can silently update the wrong table entry.
Then I would verify the reward scale, discount placement, learning rate, tie-breaking, and exploration probabilities. A deterministic one-step task is useful: if an action immediately terminates with reward 5, its target must be 5 regardless of the next-state table. A two-step chain checks whether delayed rewards propagate with exactly one discount factor per step.
I would also test invalid-action masks, copied versus aliased observations, and whether evaluation accidentally changes the learned table. Logging a few transitions and hand-computed targets is often more informative than immediately running longer experiments. Passing these fixtures validates specific mechanics, not the entire algorithm's convergence or the realism of the environment.
40. When is reinforcement learning an inappropriate choice?
Reinforcement learning is a poor fit when there is no meaningful sequential decision problem, when simpler supervised or optimization methods meet the need, or when the required exploration cannot be performed safely or affordably. A classifier with reliable labels should not become an RL project merely because the term sounds more advanced.
I would also question RL when the reward is an unreliable proxy for the actual goal, the feedback is too delayed to support learning within the budget, or the environment changes faster than data can be collected. An accurate simulator can help, but its availability does not prove that policies will transfer to the real system. Historical data may support offline learning only to the extent that its coverage and assumptions justify the proposed decisions.
My decision would compare expected benefit against interaction cost, evaluation difficulty, maintenance burden, and risk. A rule-based controller, mathematical optimizer, imitation-learning model, or contextual bandit may be the better starting point. Choosing not to use RL can be a technically sound conclusion rather than a lack of sophistication.
Reinforcement Learning Interview Questions and Answers for Intermediate
These answers explain what each update estimates, how data is reused, and where numerical or boundary-handling errors enter implementations.
41. How do n-step returns connect Monte Carlo and temporal-difference learning?
An n-step return uses n observed rewards and then bootstraps from a value estimate if the trajectory has not truly terminated. With a full n-step continuation, G_t^(n) = R_(t+1) + gamma R_(t+2) + … + gamma^(n−1) R_(t+n) + gamma^n V(S_(t+n)). One-step TD uses n = 1; using all remaining rewards through a terminal state gives a Monte Carlo return.
For rewards 2 and 3, gamma = 0.9, and a continuation value of 10, the two-step target is 2 + 0.9 × 3 + 0.9² × 10 = 12.8. If the second transition truly terminates, the target is 4.7 instead. A collection cutoff is not automatically a terminal state.
Longer returns usually reduce dependence on an inaccurate near-term bootstrap but expose the estimate to more trajectory randomness. The best horizon depends on reward delay, critic accuracy, and behavior-policy mismatch; simply increasing n does not guarantee improvement.
42. What are eligibility traces and TD(lambda)?
Eligibility traces maintain a decaying record of recently visited features or states, letting a new TD error update more than the current state. In linear accumulating-trace TD(lambda), e_t = gamma lambda e_(t−1) + x(S_t), delta_t = R_(t+1) + gamma V(S_(t+1)) − V(S_t), and w is updated by alpha delta_t e_t. A true terminal transition has zero continuation value.
Lambda controls how quickly credit fades. Lambda = 0 gives one-step TD; larger lambda gives older visits more influence. Forward-view lambda returns mix n-step returns, while backward-view traces implement related updates efficiently. Exact equivalence depends on the algorithm and whether updates occur online; ordinary accumulating traces should not be described as universally identical to every forward view.
For gamma = 0.9 and lambda = 0.8, an old trace shrinks by 0.72 each step before adding the current features. I reset episodic traces at episode boundaries and distinguish accumulating, replacing, and true-online variants when discussing implementation.
43. What changes when a value function uses function approximation?
Function approximation replaces a separate value for every state with a parameterized prediction, such as V_w(s) = wᵀx(s) or a neural network. Updating one example can therefore change predictions for other states. This generalization makes large or continuous state spaces feasible, but errors can spread through shared parameters.
A semi-gradient TD update treats its bootstrap target as fixed: w ← w + alpha [r + gamma V_w(s′) − V_w(s)] ∇_w V_w(s), with a zero continuation at true termination. It is called semi-gradient because it does not differentiate the target's dependence on w. Differentiating through both sides defines a different optimization procedure, not an automatic correction.
For a linear predictor with features [1, position], an update changes the intercept and slope across many positions. Feature scaling, coverage, and learning rate consequently matter. Tabular convergence statements cannot simply be transferred to nonlinear networks, especially with off-policy data and bootstrapped targets.
44. What is the deadly triad in reinforcement learning?
The deadly triad is the combination of function approximation, bootstrapping, and off-policy learning that can make value learning unstable or divergent. Function approximation couples predictions across states; bootstrapping uses those predictions as targets; off-policy sampling can emphasize states differently from the target policy's visitation distribution.
The issue is not that all three ingredients always fail. DQN deliberately combines them and often works, using engineering measures such as replay and target networks. Those measures reduce important sources of instability but do not establish a general convergence guarantee for arbitrary nonlinear models and environments.
I diagnose a suspected triad problem by tracking value magnitudes, TD errors, gradient norms, coverage, and target drift. A bounded-reward task with gamma below one gives a useful scale check: if absolute rewards are at most R_max, valid discounted values cannot exceed R_max/(1 − gamma) in magnitude. Violations suggest estimation problems, although staying within that bound is not proof that the policy or values are correct.
45. How does DQN turn Q-learning into a neural-network algorithm?
DQN trains a neural network Q_theta(s, a) to estimate discrete-action values using a Q-learning target. It samples transitions from replay and compares the selected action value with y = r + gamma (1 − terminated) max_a′ Q_target(s′, a′). The target is detached from the gradient, and a separate target network changes more slowly than the online network.
Python example
This fixture checks target arithmetic, not learning. A batch loss commonly uses squared error or Huber loss, while an exploratory behavior policy collects data. DQN's maximization is straightforward for a small discrete action set. A standard output head cannot directly enumerate an uncountable continuous action space; discretization or another algorithm changes that design problem.
46. Why does experience replay help, and when can it hurt?
Experience replay stores past transitions and samples them for additional updates. It improves data reuse and reduces the strong temporal correlations of training only on consecutive observations. It also separates the rate of environment interaction from the rate of optimization, which is useful when collecting experience is costly.
Replay does not turn old experience into fresh on-policy data. A transition collected by yesterday's exploratory policy can still support an appropriate off-policy update, but ordinary PPO cannot reuse it indefinitely without changing its assumptions. Very old data may also poorly represent a changing environment or current policy's useful regions.
I would choose buffer size and update-to-data ratio together. A tiny buffer can overfit recent events; a huge buffer can retain irrelevant behavior. Sampling must preserve enough metadata to interpret episode endings and observations correctly. For recurrent agents, independent single transitions may discard necessary history, so sequence sampling and hidden-state handling become part of the replay design.
47. Why does DQN use a separate target network?
A target network makes the bootstrapped regression target change more slowly than the network being optimized. Without it, improving Q_theta(s, a) also immediately changes the values used on the target side of many other updates, creating a moving-target feedback loop.
A hard update copies online parameters to the target every chosen number of optimization steps. A soft update instead uses theta_target ← (1 − tau) theta_target + tau theta_online. Here tau is the fraction of new online parameters, so a small tau means slow tracking; some libraries use an oppositely named retention coefficient, which I verify before configuring it.
For a scalar target parameter of 4, online parameter of 10, and tau = 0.01, the new target is 4.06. A slower target may improve stability but can lag important changes. Target networks are a practical stabilizer, not proof of convergence, and target calculations should normally execute without accumulating gradients.
48. How does Double DQN reduce maximization bias?
Double DQN separates next-action selection from next-action evaluation. The online network chooses a* = argmax_a Q_online(s′, a), and the target network evaluates that action: y = r + gamma (1 − terminated) Q_target(s′, a*). Ordinary DQN instead takes the maximum directly over target-network values.
Python example
The example isolates the target difference; it does not establish that the smaller estimate is the true value. Taking a maximum over noisy estimates can preferentially select positive estimation errors. Decoupling the two roles reduces that tendency, although correlated networks and approximation errors remain. Double DQN is distinct from maintaining two independently trained critics and always taking their minimum, as used in some continuous-control algorithms.
49. What does a dueling Q-network represent?
A dueling network separates a shared representation into a state-value stream and an action-relative stream before combining them into Q-values. A common aggregation is Q(s, a) = V(s) + A(s, a) − mean_b A(s, b). Subtracting the mean removes the arbitrary constant shift that would otherwise make the decomposition non-identifiable.
For V = 5 and raw advantage outputs [1, 3], the mean is 2 and the resulting Q-values are [4, 6]. The raw advantage head is an architectural quantity; it is not automatically a perfectly estimated policy advantage Q_pi − V_pi. The aggregation defines how that head is interpreted.
This architecture can share information when many actions have similar effects and state quality matters more than fine action differences. It does not change the environment, exploration policy, or target equation by itself. It can be combined with Double DQN or replay improvements, but its benefit must be tested on the actual task rather than assumed.
50. How does prioritized experience replay change sampling and correction?
Prioritized replay samples transitions non-uniformly, often using a priority based on absolute TD error plus a small positive offset. With priorities p_i and exponent alpha, P(i) = p_i^alpha / sum_j p_j^alpha. Alpha = 0 gives uniform sampling; a larger alpha concentrates updates on high-priority items.
Because the sampling distribution changes, a common correction weight is w_i = [1/(N P(i))]^beta, often normalized for numerical scale. Beta = 1 supplies the full inverse-probability correction for the corresponding uniform-buffer expectation when probabilities and weights are used correctly. Beta below one deliberately leaves partial correction; max-normalization rescales the update.
A large TD error can indicate useful surprise, but it can also indicate noise or an outlier the network cannot fit. I retain nonzero sampling probability, refresh priorities after updates, and monitor sample concentration. Replay importance weights correct replay sampling bias; they are not the same as target-policy/behavior-policy action likelihood ratios used in off-policy evaluation.
51. What does distributional reinforcement learning predict?
Distributional RL predicts a distribution of returns rather than only the expected return. Two actions can have the same mean but very different outcomes: one always returns 5, while another returns 0 or 10 with equal probability. A scalar Q-value represents both by 5; a return distribution can distinguish their variability.
Methods may represent probabilities on fixed atoms or estimate return quantiles. The prediction target must propagate both reward and discounted future-return uncertainty, with a projection or regression objective appropriate to the representation. For expected-return control, actions can still be selected using the distribution's mean.
A learned return distribution is not automatically calibrated epistemic uncertainty about the model. It can capture environmental and policy randomness while remaining confidently wrong outside the training distribution. Nor does learning a distribution automatically make the policy risk-sensitive: the selection objective must explicitly use an appropriate risk criterion. I evaluate tail predictions and the decision rule separately before making a reliability claim.
52. What is the policy gradient theorem useful for?
The policy gradient theorem expresses the derivative of expected return using policy log-probability gradients and action values, avoiding explicit differentiation of the environment's transition dynamics. For a finite episodic discounted start-state objective J = E[sum_t gamma^t R_(t+1)], one valid form is ∇J = E[sum_t gamma^t ∇log pi_theta(A_t|S_t) Q_pi(S_t,A_t)]. Other formulations absorb the discount weighting into a state-visitation measure.
The log derivative says how changing parameters changes the probability of an action. Multiplying it by a return or advantage estimate increases probability for actions with favorable outcomes and decreases it for unfavorable ones. The gradient still depends on the states visited under the policy; it is not an arbitrary supervised dataset objective.
I state the return objective and sampling convention before presenting the formula. Omitting an outer gamma^t while claiming this exact finite-trajectory discounted objective can create a mismatch. The theorem also does not eliminate the variance or sample cost of estimating its expectation.
53. How does REINFORCE estimate a policy gradient?
REINFORCE estimates a policy gradient using sampled returns from complete trajectories. For each action, I compute the reward-to-go G_t and multiply it by ∇log pi_theta(A_t|S_t), with the time weighting required by the stated objective. The environment is sampled, not differentiated.
For an undiscounted finite episode, an illustrative minimization loss is minus the sum of log probabilities times detached returns:
Python example
For the discounted start-state objective, include gamma^t outside the reward-to-go term as well. REINFORCE's Monte Carlo estimate avoids bootstrap bias under correct on-policy sampling, but can have high variance when rewards are noisy or delayed. A batch of trajectories, a baseline, and careful reward accounting can help. Normalizing returns is a practical modification, not a blanket proof that every resulting estimator remains exactly unbiased.
54. Why can an action-independent baseline reduce variance without bias?
An action-independent baseline changes the variance of a policy-gradient estimate without changing its conditional expectation, provided it is treated correctly in the actor update. For a fixed state, E_a[∇log pi(a|s) b(s)] = b(s) ∇sum_a pi(a|s) = 0. Therefore replacing Q with Q − b(s) leaves the expected score-function gradient unchanged.
A learned state-value function is a useful baseline because it removes the return attributable to being in a favorable state rather than choosing a particularly good action. If a state's usual return is 20, an action yielding 18 should not necessarily be reinforced merely because 18 is positive.
The baseline must not secretly depend on the sampled action in a way that invalidates the cancellation. In standard actor–critic code, I detach the advantage for the actor loss; otherwise gradients can flow through the critic or shared computation in unintended ways. A learned baseline's estimation process can introduce additional finite-sample dependencies, so the elementary identity is not a guarantee for every implementation.
55. What does an actor–critic algorithm do?
An actor–critic algorithm learns both an action-selection policy and a value estimate used to train that policy. The actor represents pi_theta or a deterministic action mapping; the critic estimates V, Q, or another return-related quantity. The critic supplies more immediate learning signals than waiting for an entire episode.
In a simple on-policy state-value version, delta = r + gamma V(s′) − V(s) is a one-step advantage estimate, with the continuation removed at true termination. The actor can ascend log pi(a|s) times a detached delta, while the critic minimizes a value-prediction error. These are separate objectives even if the networks share a feature encoder.
Bootstrapping can reduce variance and improve data efficiency, but critic errors can bias policy updates. I monitor both components: decreasing value loss does not prove improving policy return, especially when the target itself moves. Actor–critic is a family, not a single on-policy algorithm; PPO, DDPG, and SAC make different policy, sampling, and objective choices.
56. How do advantage estimates guide a policy update?
The advantage A_pi(s, a) = Q_pi(s, a) − V_pi(s) measures how an action compares with the policy's usual outcome from that state. A positive advantage supports increasing the action's probability; a negative advantage supports decreasing it. The comparison is relative to a specific policy, not a universal ranking of actions.
If Q_pi(s, left) = 8 and V_pi(s) = 6, the left action has advantage 2. This does not mean its immediate reward is 2: Q includes future outcomes. Under the exact policy distribution, action advantages average to zero in a state, although finite-sample estimates need not.
Practical choices include Monte Carlo return minus V, a one-step TD residual, and GAE. They trade bootstrap error against trajectory variance. I keep advantage estimates fixed during the relevant policy update and avoid treating a normalized batch advantage as a value target: normalization changes its numerical meaning. Correct state, action, and episode indexing is as important as the estimator's name.
57. How does generalized advantage estimation work?
Generalized advantage estimation forms an exponentially weighted sum of TD residuals: A_hat_t = delta_t + gamma lambda delta_(t+1) + (gamma lambda)^2 delta_(t+2) + …. It interpolates between a short-horizon, critic-dependent estimate and a longer-horizon estimate with more sampled-return variability. In a finite rollout, the sum ends at the available boundary and uses the appropriate final bootstrap value.
For residuals [1, 2], gamma = 0.9, and lambda = 0.8, the first estimate is 1 + 0.72 × 2 = 2.44. This arithmetic does not prove that the critic is correct; residual bias can propagate through the sum.
Python example
This example assumes one uninterrupted rollout segment. Episode-boundary masks must be added when multiple episodes share a batch.
58. How should GAE handle termination, truncation, and rollout boundaries?
GAE needs separate concepts for bootstrapping a transition and carrying an advantage estimate across a trajectory boundary. A true terminal state has zero bootstrap value. An external truncation can bootstrap from the actual final observation, but recursion must not connect that episode to the next reset episode.
Let b_t = 1 − terminated_t and c_t = 0 whenever the next stored transition belongs to another episode. Then delta_t = r_t + gamma b_t V(final_next_obs_t) − V(s_t), and A_t = delta_t + gamma lambda c_t A_(t+1). At an artificial rollout end, initialize the unavailable future advantage carry to zero but retain the transition's valid bootstrap.
For a time-limited continuing task, setting both masks to zero discards useful continuation value; setting both to one can leak a reset episode's rewards backward. If the horizon is genuinely part of the task, reaching it is termination and remaining time should be represented when required for the Markov property. I test these cases separately.
59. How do A2C and A3C differ?
A3C uses asynchronous actor–learner workers that interact with environments and update shared parameters without waiting for a synchronized batch. A2C uses a synchronized collection-and-update pattern, typically gathering rollout segments from several environments before one coordinated update. Both are commonly advantage actor–critic methods with value and entropy terms.
Asynchrony can diversify data collection and avoid waiting for every worker, but introduces stale parameters and nondeterministic update ordering. Synchronization is often easier to batch efficiently and reason about, although slow environments can create a straggler cost. Neither execution style universally wins on sample efficiency or wall time.
I would specify the number of environments, rollout length, update batch size, optimizer, and hardware before comparing them. If four workers each collect five transitions, the synchronized update receives twenty transitions; changing the worker count without adjusting other settings changes more than parallelism. Episode boundaries still need correct bootstrapping and recurrent-state resets, regardless of the worker coordination strategy.
60. What does the PPO clipped objective optimize?
PPO's clipped surrogate limits the incentive to move sampled action probabilities too far in the advantage-favored direction. With r_t(theta) = pi_theta(a_t|s_t)/pi_old(a_t|s_t), it maximizes the batch average of min[r_t A_hat_t, clip(r_t, 1 − epsilon, 1 + epsilon) A_hat_t]. The stored behavior log probabilities and advantage estimates are fixed during the update.
For A = 2, r = 1.4, and epsilon = 0.2, the two terms are 2.8 and 2.4, so the contribution is 2.4. For A = −2 and r = 0.6, they are −1.2 and −1.6, so the contribution is −1.6. The sign-sensitive minimum matters; clipping the ratio alone is not the same objective.
Python example
Implementations usually minimize the negative surrogate alongside separately weighted value and entropy terms.
61. Does PPO clipping guarantee a trust region or monotonic improvement?
No. PPO clipping modifies the sampled surrogate objective; it does not impose a hard bound on every probability ratio, the policy's KL divergence, or actual return degradation. Shared parameters, unsampled actions, value-loss gradients, and multiple optimization epochs can still produce a large policy change.
The clipped term removes a particular incentive once a sampled action has moved sufficiently in the favored direction. It does not project parameters back into a feasible trust region. For a positive advantage, increasing an already-too-large ratio may give no further surrogate gain, but other samples can still push the same parameters.
I therefore monitor approximate KL divergence, clip fraction, entropy, value behavior, and held-out episodic outcomes. Some implementations stop an update early when KL exceeds a configured threshold, which adds a useful operational guard but is not a universal performance proof. I describe PPO as a practical conservative-update heuristic rather than claiming its clipping guarantees monotonic improvement in noisy finite-data training.
62. What should a practical PPO update keep fixed and monitor?
A PPO update should preserve the rollout's behavior-policy log probabilities, sampled actions, observations, and chosen advantage and return targets while optimizing on that batch. Recomputing the denominator with the changing policy would collapse the likelihood ratio toward one and destroy its meaning.
I compute ratios as exp(new_log_prob − old_log_prob), check their finiteness, and use the joint action log probability for multidimensional actions. Before the first update, identical old and current policies should give ratios near one. A sudden discrepancy often reveals preprocessing drift, dropout, incorrect action transforms, or mismatched recurrent state.
Useful diagnostics include approximate KL, clip fraction, policy entropy, actor and critic losses, explained variance, gradient norms, and actual episode outcomes. I also track how many epochs and minibatch updates use each rollout. Advantage normalization and value clipping are implementation choices that must be stated, not confused with the defining clipped-policy objective. Once the batch has served its intended update cycle, I collect new on-policy experience.
63. How does TRPO differ from PPO?
TRPO optimizes a policy surrogate while approximately constraining the average KL divergence from the old policy. Practical implementations use a local approximation, conjugate-gradient computations involving curvature, and a line search. PPO commonly uses a clipped surrogate with ordinary minibatch optimization instead of that constrained update procedure.
Both seek useful policy improvement without overly large changes, but the mechanisms differ. A clipped probability-ratio objective is not the same as solving a KL-constrained problem. TRPO's theoretical improvement argument also relies on assumptions and bounds that practical approximations do not satisfy exactly; an empirical update is not an unconditional monotonic-return guarantee.
I would compare them in terms of implementation complexity, optimization cost, batch requirements, sensitivity, and measured task performance. PPO may be simpler to integrate with common training pipelines, while TRPO makes the explicit trust-region idea clearer. Neither name replaces checking the advantage estimator, policy distribution, environment endings, and whether the chosen objective matches the task's return definition.
64. How does a deterministic policy gradient work?
A deterministic policy gradient updates a continuous-action policy mu_theta(s) by differentiating a critic through the action: a typical sampled actor direction is E_s[∇_theta mu_theta(s) ∇_a Q_phi(s,a) evaluated at a = mu_theta(s)]. This avoids summing over all actions and lets the actor exploit a differentiable Q-function.
The chain rule has two parts: how parameters change the chosen action, and how the critic predicts that changing the action changes return. For Q(s,a) = −(a − 2)^2, the action derivative is −2(a − 2), which points toward 2. A learned critic can supply a misleading direction when it is inaccurate.
Replay-based implementations optimize a surrogate over sampled replay states; the exact relationship to a desired start-state objective depends on visitation assumptions and corrections. During the actor update, I retain the gradient through Q with respect to the action while not updating critic parameters. Exploration generally requires additional noise or another exploratory behavior policy.
65. What are the main components and failure modes of DDPG?
DDPG is an off-policy continuous-control actor–critic method combining a deterministic actor, an action-value critic, replay, and slowly updated target networks. A common critic target is r + gamma (1 − terminated) Q_target(s′, mu_target(s′)); the actor increases Q(s, mu(s)). Noise added during collection supplies exploration around the deterministic action.
The actor can exploit errors in its learned critic, especially where replay provides little support. Overestimated values, poor exploration, sensitive learning rates, and inappropriate action scaling can destabilize training. A smooth-looking value loss is not sufficient evidence that the behavior is improving.
I verify action bounds and normalization, target-network update conventions, replay coverage, and correct gradient routing before tuning architecture. For an actuator with physical range [−10, 10], an internal tanh output in [−1, 1] needs an explicit consistent transform. DDPG is an important baseline and conceptual building block, not a default guarantee of good performance for every continuous-action task.
66. How does TD3 address errors in deterministic actor–critic learning?
TD3 addresses several approximation-error problems with three linked changes: twin critics with a minimum target, delayed actor updates, and target-policy smoothing. Its critic target evaluates a clipped noisy target action using the smaller of two target Q-values. The actor and target networks are updated less frequently than the critics.
Taking the minimum discourages optimistic target errors, although it can introduce underestimation and does not remove all critic bias. Smoothing adds bounded noise to the target action before evaluating it, making it harder to exploit a narrow, erroneous peak in the learned value surface. That target noise is distinct from exploration noise used to collect transitions.
For target critics predicting 7 and 10 at the smoothed action, the continuation estimate is 7, not their mean. I test action clipping after smoothing and verify update frequencies in optimizer-step units. TD3's original benchmark improvements motivate testing it, but they do not establish superiority for every reward scale, environment, or compute budget.
67. What objective does soft actor–critic optimize?
Soft actor–critic optimizes an entropy-regularized return, rewarding both task reward and policy entropy. A common form is E[sum_t gamma^t (r_t + alpha H(pi(·|s_t)))], where alpha controls the reward–entropy trade-off. SAC is off-policy and uses a stochastic actor with replay-based critic learning.
Higher entropy discourages prematurely collapsing to a narrow action distribution, but random behavior is not inherently useful or safe. The entropy term changes the optimization objective; it is not merely a trick that leaves the original reward-only optimum unchanged. Temperature selection or adaptation therefore matters, especially when reward scale changes.
I distinguish the original SAC formulation, which included a separate value network, from common later implementations using target Q-networks without a separate value network. I also specify whether evaluation samples actions or uses a deterministic representative action. Those choices can change observed performance. SAC is well suited to many continuous-action experiments, but the task and constraints still determine whether it is appropriate.
68. How are the SAC critic and actor losses constructed?
In a common twin-Q SAC implementation, the detached critic target is y = r + gamma (1 − terminated) [min_i Q_target_i(s′,a′) − alpha log pi(a′|s′)], with a′ sampled from the current actor. Each critic minimizes error against y. The actor minimizes E[alpha log pi(a|s) − min_i Q_i(s,a)] over sampled replay states and reparameterized policy actions.
The signs matter: minimizing the actor loss favors higher predicted return and higher entropy, because expected negative log probability is entropy. During the actor update, gradients must pass through the sampled action and Q's action input, while critic parameters are not stepped by that loss. The critic target is computed without gradients.
For min Q = 5, log pi = −2, and alpha = 0.1, the soft continuation is 5.2. Continuous densities can also exceed one, so log density is not always negative. I check the actual distribution and action-transform correction rather than assuming every entropy bonus is positive.
69. How does a squashed Gaussian policy handle bounded continuous actions?
A squashed Gaussian policy samples an unconstrained latent action u from a Gaussian and maps it through a = tanh(u), then optionally rescales to environment bounds. The transformed distribution is not Gaussian. Its log density must include the change-of-variables correction: log pi(a|s) = log p(u|s) − sum_j log(1 − tanh(u_j)^2), plus any action-scale Jacobian term.
For a multidimensional action, I sum component log densities to obtain one joint log probability per sample. Ignoring the squash correction gives incorrect entropy and likelihood calculations. Near saturation, direct subtraction can be numerically unstable, so practical implementations use stable equivalent formulas or well-defined numerical safeguards.
I store the latent sample when necessary rather than reconstructing it from heavily clipped actions. Arbitrarily clipping a Gaussian action is not equivalent to a tanh transformation: clipping creates boundary mass and changes the likelihood model. Correct bounds, scaling, sampling, and log probabilities must agree in both collection and training.
70. What does entropy regularization change about learning?
Entropy regularization rewards a policy for retaining uncertainty over actions, usually by adding beta H(pi(·|s)) to a maximization objective. In a minimization loss, the corresponding entropy term has a negative sign. This can discourage early collapse and support exploration, but the benefit depends on the task and coefficient.
For a two-action categorical policy, [0.5, 0.5] has entropy log 2, while [1, 0] has zero entropy using the limiting convention 0 log 0 = 0. A large coefficient can keep the agent random even when one action is clearly better. Entropy over unsafe actions is not a substitute for enforcing constraints.
For continuous actions, differential entropy depends on coordinate scale and can be negative. Rescaling actions changes its numerical value, so coefficients and target entropy need an explicit convention. I also distinguish a small auxiliary entropy bonus in an on-policy method from SAC's soft-value formulation, where entropy is incorporated consistently into the critic's return target.
71. How does importance sampling support off-policy learning?
Importance sampling reweights behavior-policy data to estimate an expectation under a different target policy. For a sampled action in a fixed state, rho = pi(a|s)/b(a|s). A trajectory-level ratio multiplies the relevant action ratios when initial-state and environment dynamics distributions match and thus cancel.
The key support condition is that the behavior policy assigns positive probability wherever the target policy assigns positive probability for the quantity being estimated. No finite weight can recover an action or trajectory that the data-collection process could never produce. Logged propensities must also describe the actual behavior policy, including exploration or action filtering.
For an action chosen with behavior probability 0.25 and target probability 0.5, the ratio is 2. This doubles its contribution to the corresponding weighted expectation; it does not mean the observed return itself has doubled. Importance sampling is a change-of-distribution tool, not a guarantee of low variance, reliable coverage, or valid causal evaluation from arbitrary logs.
72. Why can trajectory importance sampling have high variance?
Trajectory importance sampling can have high variance because it multiplies many random likelihood ratios. Even modest stepwise ratios can produce very large or tiny products over a long horizon. Rare trajectories with large weights may dominate an estimate, leaving the effective information much smaller than the nominal dataset size.
If ten successive sampled actions each have target-to-behavior ratio 2, the trajectory weight is 2^10 = 1024. That arithmetic illustrates the potential scale, not the probability of such a trajectory. Computing log weights prevents some numerical overflow but does not solve the statistical variance problem.
Per-decision weighting avoids multiplying ratios for decisions that occur after a reward contribution. Weighted estimators, clipping, and specialized off-policy return operators can further manage variance, but may introduce bias or require additional assumptions. I report weight concentration, support concerns, and uncertainty instead of presenting one noisy weighted average as a dependable policy ranking. A near-zero denominator in self-normalized estimation is a warning, not a value to silently patch.
73. What is the difference between ordinary and weighted importance sampling?
Ordinary importance sampling averages weighted outcomes: with N trajectories, the estimate is sum_i w_i G_i / N. Under the relevant support and sampling assumptions, it is unbiased for the target expectation, though its variance may be large or even fail to be finite. Weighted, or self-normalized, importance sampling divides by sum_i w_i instead.
Python example
The ordinary estimate can lie outside the observed return range; the normalized estimate with nonnegative weights lies within it. Self-normalization is generally biased in finite samples, although it can be consistent under suitable assumptions and often reduces practical variance. I handle a zero total weight explicitly and do not call the normalized estimator universally better: coverage, tail behavior, horizon, and uncertainty still matter.
74. Why are off-policy multi-step corrections needed?
Off-policy multi-step returns need care because intermediate rewards and actions were generated by the behavior policy, not necessarily the target policy. One-step Q-learning handles its next-action target with a greedy maximum, but extending that target through several behavior-selected actions changes what the sampled return represents.
Naively combining many behavior rewards with a final target-policy bootstrap can therefore create bias relative to the intended target-policy value. Importance sampling can correct the action-distribution mismatch, but long products may have high variance. Algorithms such as Retrace and V-trace use structured, often clipped corrections to obtain useful stability and learning properties under their stated assumptions.
The clipping details matter: truncating a ratio is not always an innocuous numerical implementation change, and different operators can imply different fixed points or correction properties. I name the operator, specify which ratios weight TD errors and trace propagation, and test it on a small MDP. I would not describe arbitrary replayed n-step returns as unbiased on-policy estimates.
75. How should a replay buffer represent terminal and truncated transitions?
A replay buffer should preserve the transition's true next observation, reward, action, and separate termination and truncation signals, or equivalent metadata sufficient to reconstruct their meaning. A single reset flag is often insufficient because ending data collection is different from reaching a terminal state of the task.
For bootstrapped targets, I suppress continuation on true termination. For an external time-limit truncation in a continuing task, I use the final observation before reset and retain continuation. With automatic-reset vector environments, the observation returned to the collector may be the new episode's initial observation under the configured reset mode; the collector must retrieve and store the correct final observation according to that API.
Python example
I also preserve episode IDs for sequence sampling so traces or recurrent states cannot leak across resets.
76. How can recurrent policies learn from partially observed trajectories?
A recurrent policy uses a hidden state summarizing observation history, allowing actions to depend on information that is not visible in the current observation. This can help in a POMDP, such as inferring velocity from successive positions. It does not guarantee that the learned hidden state is a sufficient belief state.
Training must preserve temporal order. I sample contiguous sequences, reset hidden state at episode boundaries, mask padded timesteps, and use a defined truncated-backpropagation window. With replay, a stored hidden state may be stale because network parameters have changed; a burn-in prefix can reconstruct a more current state before the optimized sequence.
The action distribution used for a stored likelihood must match the observation history and hidden state used during collection. Otherwise PPO ratios can be wrong even if network weights appear identical. I test two sequences with the same current observation but different histories, and verify that resetting one environment does not erase another environment's state in a vectorized batch.
77. How do observation and reward normalization affect an RL implementation?
Observation normalization changes the scale of inputs, which can make optimization easier when features have very different magnitudes. Reward normalization or scaling changes target magnitudes and interacts with learning rates, value losses, clipping thresholds, and entropy coefficients. These are part of the algorithm configuration, not harmless preprocessing details.
For positive constant reward scaling in a reward-only discounted MDP, the ideal policy ranking is unchanged, but the behavior of a finite neural optimizer can change substantially. With an entropy-regularized objective, scaling rewards without adjusting the temperature changes the relative reward–entropy trade-off. Reward clipping can change the task's preference ordering entirely.
I save running statistics with the policy and freeze their updates during evaluation unless the evaluation protocol explicitly specifies adaptation. Training and evaluation must use the same observation transform. I report returns in original environment units for interpretability and distinguish per-step reward statistics from discounted-return statistics. Updating normalization using held-out evaluation trajectories can leak information into a supposedly fixed policy test.
78. Which numerical and gradient checks catch common actor–critic bugs?
I start with shapes, finite values, gradient routing, and tiny deterministic calculations before any long training run. A common silent error is broadcasting a value tensor of shape [batch, 1] against a target of shape [batch], producing a [batch, batch] error matrix instead of one error per transition. I assert intended shapes explicitly.
For stochastic actors, I verify that log probabilities sum over action dimensions but retain the batch dimension. I detach actor advantages and critic targets where required, while preserving the action derivative for reparameterized or deterministic actor updates. Gradient clipping cannot fix a fundamentally incorrect loss sign.
I also check terminal targets, reward scales, action bounds, initial PPO ratios near one, target-network copies, and whether gradients reach only the intended parameter groups. For categorical policies, a stable log-softmax avoids manually taking logarithms of tiny probabilities. Unit tests establish local arithmetic and implementation contracts; only properly controlled learning experiments can establish whether the complete method actually improves return.
79. How do action masks interact with policy probabilities and training?
An action mask removes invalid actions from the policy's support before sampling, typically by setting their categorical logits to negative infinity and renormalizing over valid actions. The same masked distribution must be used to compute training log probabilities and entropy. Sampling from one distribution and evaluating its likelihood under another gives incorrect policy updates.
For logits [0, 0, 0] and mask [valid, invalid, valid], the resulting probabilities should be [0.5, 0, 0.5]. I reject an all-invalid mask or define an explicit valid fallback action; applying softmax to all negative-infinity logits produces invalid values. Entropy calculations must handle zero-probability entries safely.
Masks should represent genuine action feasibility, not secretly encode privileged future information. If feasibility depends on hidden history, that dependency belongs in the environment or policy-state design. For off-policy methods, target actions must respect the next state's validity mask as well. Masking can prevent structurally invalid choices, but it does not prove that all remaining choices satisfy safety or business constraints.
80. How would you choose between DQN, PPO, and SAC for a bounded experiment?
I choose among DQN, PPO, and SAC by action space, interaction cost, implementation constraints, and the experiment's objective rather than a universal ranking. DQN is a natural baseline for manageable discrete actions. PPO supports stochastic policies across several action-space designs and uses recent on-policy rollouts. SAC is a common off-policy choice for continuous control with reusable replay data.
I first define a bounded simulator task, reward and termination semantics, evaluation seeds, interaction budget, and compute budget. Then I compare a simple baseline and one or two suitable algorithms using the same evaluation protocol. Environment steps and wall time answer different efficiency questions, so I report both where measured.
A method requiring fresh rollouts may be unsuitable when interactions are expensive; a replay method may struggle with poor coverage or changing dynamics. Hyperparameter tuning must not consume the held-out test set. A local numerical fixture verifies formulas only; selecting an algorithm for deployment requires additional task-specific performance, robustness, and constraint evidence that this illustrative comparison does not supply.
Reinforcement Learning Interview Questions and Answers for Experienced
These answers distinguish a plausible algorithmic design from evidence that a policy is useful, robust, and appropriate for its operating constraints.
81. What does model-based reinforcement learning add beyond a model-free policy?
Model-based RL uses an explicit model of transitions, rewards, or both to help choose or improve actions. A model-free value function predicts return without explicitly predicting the next state. Either approach may use neural networks, and hybrid systems can contain both components.
For example, a controller can learn p(s′ | s, a), simulate several action sequences, and choose one with high predicted discounted reward. Alternatively, it can use imagined transitions to train a value function that acts cheaply at deployment. The model therefore changes how experience is reused; it does not automatically eliminate the need for real observations.
I would compare environment interactions, computation, model error, and task constraints. Model-based learning can save expensive interactions, but planning may be costly and exploit model mistakes. A useful comparison holds the data budget and evaluation protocol fixed rather than assuming a model-based method is universally more sample-efficient.
82. How does Dyna combine learning from experience with planning?
Dyna interleaves direct value updates from real transitions, updates to an environment model, and planning updates using simulated transitions from that model. In tabular Dyna-Q, both real and imagined samples can use the same Q-learning backup. The important distinction is where the transition came from.
Python example
More planning steps reuse each real observation more heavily, but an incorrect model also gets amplified. A last-observation model is appropriate only for the stated deterministic illustration, not arbitrary stochastic dynamics. I would store or estimate transition distributions for stochastic tasks and account for stale models when the environment changes.
83. How would you implement model-predictive control with a learned dynamics model?
Model-predictive control repeatedly plans over a limited horizon, executes only the first action, then replans from the new observation. With a learned dynamics model, I score candidate sequences using predicted rewards, constraints, and optionally a terminal value estimate. Replanning reduces dependence on a single long open-loop prediction.
Python example
The planning budget must fit the control deadline. I would validate state estimation, actuator limits, delay, and fallback behavior before considering real use. Feasibility under an approximate model is not proof of physical safety; independent action checks and uncertainty-aware margins remain necessary. A longer horizon can improve foresight while increasing computation and exposure to model error.
84. Why can an accurate one-step model still produce a poor policy?
One-step predictive accuracy is averaged over a data distribution, whereas a planner searches for actions that maximize predicted return. The selected actions can lead into poorly covered states, where small errors compound and the planner exploits optimistic predictions. Low mean squared prediction error therefore need not imply accurate policy ranking.
Suppose a model slightly underestimates energy use for an unusual high-speed maneuver. Repeated planning may favor that maneuver precisely because of the error, shifting the next observations away from the training data. Validation only on ordinary trajectories misses this failure.
I would evaluate multistep predictions on relevant trajectories, reward and termination predictions, uncertainty calibration, and actual policy performance in bounded tests. Short model rollouts from real states, ensembles, conservative objectives, and refreshed data can help. None removes the need to check whether the model remains reliable where the improved policy actually goes.
85. How do world models handle partially observed environments?
A world model can maintain a latent state that summarizes observation and action history, rather than treating the current observation as a fully observed state. A recurrent latent dynamics model can predict future latent states, rewards, and continuation probabilities, allowing policies to learn from imagined trajectories.
For example, one camera image may not reveal an object's velocity. A history-dependent latent representation can encode information from successive images and previous actions. In a POMDP, the exact Bayesian belief over hidden states is a sufficient statistic when the model is correct; a learned latent state is only an approximation to that ideal.
I would test whether the representation preserves information needed for control, not just whether reconstructed images look convincing. Latent rollouts can drift, omit rare hazards, or misestimate termination. Imagination is a computational tool for learning and planning, not evidence that imagined successes transfer unchanged to the real environment.
86. What makes offline reinforcement learning different from ordinary off-policy learning?
Offline RL learns a policy from a fixed dataset without collecting fresh interactions during that learning stage. Ordinary off-policy learning also uses data from other policies, but it may keep interacting and correct mistaken value estimates through new experience. Offline learning lacks that repair mechanism.
The central problem is distribution shift: maximizing a learned Q-function can select actions whose outcomes are weakly represented or absent in the dataset. Bootstrapping then propagates unsupported predictions. Having a large replay buffer alone does not make a standard off-policy algorithm suitable for a strictly offline task.
I would examine how the data was collected, whether relevant actions have support, how termination is recorded, and whether rewards match the intended objective. Conservative value estimation, policy constraints, and imitation baselines address parts of the problem. They cannot establish the value of arbitrary unseen behavior without additional assumptions or evidence.
87. How do you assess dataset coverage before attempting offline RL?
I assess coverage at the state, action, and trajectory levels, because plentiful individual observations do not guarantee coverage of the sequences a new policy will induce. I examine behavior-policy mixtures, action frequencies or densities conditional on context, rare outcomes, missing fields, and whether important operating regions are represented.
For a discrete example, a logging policy that never chooses action B in a certain state provides no direct evidence about B there. A target policy selecting B violates the overlap requirement for importance-sampling evaluation. In continuous spaces, exact action repetition is not required, but support and smoothness assumptions become particularly important.
I would also inspect timestamp alignment, interventions, censored episodes, and whether logs include all variables affecting decisions and outcomes. A coverage report should identify unsupported regions explicitly. Model generalization may be useful, but a confident neural prediction should not be mislabeled as empirical coverage.
88. How does Conservative Q-Learning reduce unsupported value optimism?
Conservative Q-Learning adds a value regularizer that discourages high Q-values for actions outside the dataset relative to recorded actions. A representative discrete-action term is α times E_s[log Σ_a exp Q(s,a) − E_{a~D}[Q(s,a)]], added to a Bellman-error objective. It pressures the model against unsupported optimistic action values.
This addresses a characteristic offline-RL failure: a policy selects an unobserved action because approximation error makes it look unusually good. The conservatism coefficient controls a trade-off; too little may leave optimism, while too much may suppress worthwhile improvement.
CQL's lower-bound results depend on the formulation and theoretical assumptions. I would not claim that every neural implementation yields a certified lower bound for every state and action. Dataset quality, sampling approximations, optimization, and tuning matter. I would compare it with imitation and other offline baselines under the same evaluation budget.
89. How does Implicit Q-Learning avoid querying unseen actions during training?
Implicit Q-Learning estimates an upper expectile of dataset-action Q-values as a state value, then backs up that value into Q and extracts a policy through advantage-weighted behavior cloning. Its training targets avoid maximizing Q over newly sampled, potentially unsupported actions.
For residual u = Q(s,a) − V(s), the expectile loss weights u² by τ when u is nonnegative and by 1−τ otherwise. Choosing τ above 0.5 emphasizes better dataset actions. Expectiles are asymmetric squared-error summaries, not quantiles.
Python example
The extracted policy can still generalize beyond the exact logged actions, so this is not an absolute deployment support guarantee. I would validate the learned policy separately and inspect advantage-weight clipping and reward scaling.
90. When is behavior cloning a better starting point than offline RL?
Behavior cloning is a strong starting point when demonstrations are already good, action coverage is narrow, rewards are unreliable, or the project needs a simple auditable baseline. It fits the behavior directly through supervised learning rather than bootstrapping long-horizon value estimates.
For discrete actions, a typical objective is −E_D[log πθ(a|s)]. This can be easier to optimize and evaluate than offline RL. However, it reproduces dataset behavior, including poor choices, and prediction errors can move the policy into states absent from demonstrations. Matching held-out action labels is not the same as achieving high closed-loop return.
I would first determine whether there is useful room to improve and enough evidence to support improvement. Offline RL is attractive when good subtrajectories can be combined or delayed outcomes distinguish actions, but it adds estimation risk. A more complex learner should demonstrate benefits over a well-tuned cloning baseline, not merely outperform an intentionally weak one.
91. How does trajectory importance sampling estimate an unseen policy's return?
Trajectory importance sampling reweights each logged trajectory by how much more or less likely it is under the target policy π than the behavior policy b. With matched environment dynamics and initial-state distribution, w = Π_t π(a_t|h_t)/b(a_t|h_t), and ordinary IS averages wG over trajectories.
If a two-step trajectory has ratios 1.5 and 0.5, its total weight is 0.75; a return of 8 contributes 6 to the numerator of the ordinary sample mean. Histories h_t are appropriate when the decision process is not fully observed.
The estimator requires support: behavior must assign positive probability wherever the target can generate relevant trajectories. Correct propensities, consistent rewards, and valid logging assumptions also matter. Ordinary IS is unbiased under these conditions but can have enormous variance. It cannot recover outcomes for actions never taken simply by assigning a very large weight to other actions.
92. Why do importance weights become unstable, and what can you do about it?
Importance weights multiply across time, so modest per-step mismatch can create extremely uneven trajectory weights. A few trajectories may dominate an estimate, while many carry almost no information about the target policy. This worsens with long horizons and weak overlap.
I inspect weight distributions and effective sample size, ESS = (Σw)²/Σw², alongside uncertainty intervals. ESS is a diagnostic, not a guarantee that bias or hidden confounding is absent.
Python example
Per-decision weighting can reduce unnecessary multiplication for earlier rewards. Self-normalization and clipping often reduce variance but generally introduce bias; clipping also changes the estimator's target behavior unless analyzed carefully. Better data coverage or a policy closer to the logger may be more defensible than presenting a stabilized estimate as reliable by default.
93. What makes an off-policy value estimator doubly robust?
A doubly robust estimator combines a learned value model with importance-weighted corrections for prediction errors. The model supplies a low-variance baseline; logged outcomes correct its mistakes when propensity ratios are valid. The robustness claim is conditional, not permission to ignore overlap or data quality.
For a one-step contextual bandit, a standard form is V̂(x) + [π(a|x)/b(a|x)](r−Q̂(x,a)), where V̂(x)=Σ_a π(a|x)Q̂(x,a). With suitable independence or cross-fitting, support, and regularity conditions, correctness of either the propensity model or the reward model can give consistency. Sequential estimators extend this idea using cumulative ratios and temporal residuals, with additional nuisance-model conditions.
I would state which estimator and assumptions I am using instead of asserting that any mixture of a critic and IS is doubly robust. If both models are badly misspecified, the result can be wrong; a small empirical variance does not prove accuracy.
94. How does fitted Q evaluation work, and where can it fail?
Fitted Q evaluation estimates the value of a fixed target policy from logged transitions. It repeatedly regresses Q(s,a) toward r + γ E_{a′~π}[Q_previous(s′,a′)], using zero future value at genuine terminal states. It then averages the target policy's estimated values over the intended initial-state distribution.
Unlike policy optimization, FQE keeps the target policy fixed while fitting the evaluator. It can avoid the extreme explicit weights of trajectory IS, but it substitutes function-approximation and coverage assumptions. Repeated backups can extrapolate into unsupported actions or states and amplify error.
I would separate evaluator fitting and policy selection where possible, inspect Bellman residuals on held-out trajectories, and compare multiple evaluator families. Small residuals on observed data do not guarantee correct value elsewhere. For external time-limit truncations, the evaluator needs the actual final observation and appropriate continuation value, not a fabricated terminal zero or the next episode's reset observation.
95. How would you decide whether offline evaluation is sufficient for deployment?
Offline evaluation is sufficient only for the decision it can support under its assumptions and the consequences of error. I would require a clearly specified target population and horizon, credible logging and overlap, uncertainty estimates, baseline comparisons, and evidence that the proposed policy stays within supported operating regions.
Agreement between IS, a model-based evaluator, and FQE is helpful but not decisive: they may share the same missing confounder or flawed reward definition. I would inspect disagreement, stress-test plausible misspecification, and include tail-risk and constraint outcomes rather than return alone.
For consequential deployment, a favorable offline estimate ordinarily supports a bounded next step, not unrestricted rollout. The next step might be simulation, shadow recommendations, human-reviewed decisions, or an explicitly authorized limited experiment with stop rules. If no safe experiment is possible and the evidence cannot rule out unacceptable harm, the correct decision can be to retain the existing policy.
96. How do you design a statistically credible RL benchmark comparison?
I predefine tasks, budgets, metrics, tuning rules, and evaluation seeds, then compare methods across independent training runs rather than reporting only the best seed. Environment interactions and wall-clock cost measure different resources, so I report both when they matter.
Evaluation episodes within one trained policy do not replace independent training runs: they quantify different sources of uncertainty. For a task suite, I would show per-task results plus an appropriate aggregate, such as an interquartile mean with stratified bootstrap intervals, and explain score normalization. Raw returns from unrelated tasks should not be averaged blindly.
I also report failure rates, constraint violations, and sensitivity to hyperparameters. A point estimate above a baseline is not automatically a meaningful improvement when intervals are wide or tuning budgets differ. The conclusion should describe the tested tasks and budget, not declare an algorithm universally superior.
97. How do you prevent evaluation leakage in an RL project?
I separate training, validation, and final testing at the level where dependence occurs. In sequential data, randomly splitting individual transitions can leak neighboring states from the same episode into multiple partitions. Time, user, machine, or trajectory-based splits may better match the intended generalization claim.
Policy selection and hyperparameter tuning must not repeatedly use the final test set. Normalization statistics, learned encoders, reward models, and simulator calibration can also leak information if fitted using held-out outcomes. For benchmark tasks, repeatedly designing around known test seeds can turn them into an informal validation set.
I would document every dataset's purpose and freeze a final evaluation protocol before the decisive comparison. In a changing environment, a forward-in-time holdout can be more realistic than a random split. No partition strategy guarantees external validity; the split must represent the deployment shifts the project claims to handle.
98. What must be recorded to make an RL result reproducible?
I record code and dependency versions, environment and wrapper versions, configuration, training and evaluation seeds, dataset hashes, reward definitions, termination semantics, preprocessing, checkpoint selection, hardware, and the actual evaluation procedure. A seed alone does not define an experiment.
For off-policy learning, replay-buffer handling and update-to-data ratio can strongly affect results. For parallel training, worker counts, batching, and asynchronous data order may matter. I preserve raw per-run metrics and failed runs rather than only a smoothed aggregate curve.
Reproducibility has levels: another run may reproduce the statistical finding without matching every floating-point operation. Nondeterministic kernels and hardware differences can prevent bitwise equality even with seeds. I would state which level was tested, how much variability remained, and whether an independent rerun was actually performed. Configuration capture is necessary evidence, not proof that the claimed performance has been reproduced.
99. How do you distinguish training instability from an environment integration bug?
I first test environment and data contracts on tiny deterministic cases before changing the algorithm. I inspect observation shapes, action bounds, reward timing, reset behavior, normalization, and terminal flags, then verify a hand-calculated target. An algorithm cannot compensate for receiving the next episode's observation as the previous transition's successor.
For Gymnasium-style interfaces, reset is required after termination or truncation, but bootstrapping is normally suppressed only for genuine task termination. A finite horizon inherent to the task is terminal and needs remaining-time information in the state; an external rollout cutoff is different.
I then track losses, gradient norms, action distributions, value scales, and performance over multiple seeds. A deterministic toy failure suggests an implementation issue, while seed-sensitive failure after the contracts pass may reflect optimization or exploration. Neither pattern is definitive, so I isolate changes and preserve failing trajectories rather than explaining every failure as RL instability.
100. How do you address the simulation-to-reality gap?
I treat simulation-to-reality transfer as a distribution-shift problem involving dynamics, observations, delays, disturbances, and action execution. A simulator that looks realistic may still model the variables important to control incorrectly. I would identify these variables before choosing a transfer method.
Domain randomization trains across ranges of plausible parameters; system identification estimates parameters from real observations; adaptation can update a model as conditions change. Their usefulness depends on whether the training variation covers the actual deployment uncertainty. Wider randomization is not automatically better if it makes the task unrealistic or overly conservative.
I would validate sensor preprocessing and control timing, test rare disturbances, and use independent safety limits with a known fallback controller. Initial real-world testing should be explicitly authorized and bounded. A high simulated return is evidence about a simulator, not a substitute for measured transfer performance or a guarantee of physical safety.
101. How do constrained MDPs represent safety requirements?
A constrained Markov decision process optimizes expected return subject to explicit expected-cost constraints, such as J_cost(π) ≤ d. Rewards represent the objective, while costs encode undesirable events or resource consumption. The discounting and averaging convention must be specified for both.
A common Lagrangian is L(π,λ)=J_reward(π)−λ(J_cost(π)−d), with λ nonnegative. The policy seeks higher L, while the multiplier grows when the constraint is violated. For example, energy use may be capped while throughput is maximized.
An expected-cost constraint is not automatically a per-trajectory or per-step safety guarantee. A low average can conceal rare severe violations, and approximate optimization or model error can invalidate theoretical guarantees. I would choose the constraint type to match the actual requirement and combine learning with independently enforced limits where violations are unacceptable.
102. Why is a reward penalty not the same as a hard safety guarantee?
A reward penalty trades one outcome against another; a hard safety constraint forbids an action or trajectory regardless of its predicted reward. If a policy can gain enough task reward, a finite penalty can still make an unsafe action optimal under the specified objective.
For example, maximizing revenue minus ten times a violation indicator can rationally accept a violation worth more than ten reward units. Increasing the coefficient changes that trade-off but does not prove all violations are impossible, especially with reward scaling, estimation error, or unseen states.
I would enforce non-negotiable limits through a validated action filter, controller, or authorization layer, while using cost-aware learning to improve behavior within those limits. Such safeguards have their own assumptions and failure modes. A shield based on an inaccurate model or missing state information is not a universal guarantee, so its coverage and fallback behavior need separate testing.
103. How can potential-based reward shaping preserve an optimal policy?
Potential-based shaping adds F(s,a,s′)=γΦ(s′)−Φ(s) to the original reward. Under the usual discounted-MDP conditions, the extra discounted terms telescope, shifting a trajectory's return by a state-dependent constant rather than changing the preferred action ordering.
For a finite trajectory ending at T, the added return is −Φ(s₀)+γ^TΦ(s_T). Setting terminal potentials to zero removes the final term; in an infinite discounted setting, bounded Φ makes the limit vanish. Arbitrary distance bonuses do not automatically satisfy this structure.
Python example
I would preserve the original reward in evaluation logs and test terminal handling. Shaping can improve learning signals without fixing poor exploration, an incorrect base objective, or partial observability.
104. How would you detect and respond to reward hacking?
Reward hacking occurs when a policy increases the measured reward through behavior that defeats the intended task. I would look for a divergence between proxy reward and independently measured outcomes, suspicious repeated behaviors, boundary exploitation, or sudden reward jumps without corresponding task success.
For example, a cleaning agent rewarded for detecting dirt might learn to revisit or generate dirt instead of leaving the room clean. Simply adding another patch can create a new loophole, so I would examine the causal link between the reward and the desired outcome.
The response is to preserve failure traces, restrict the affected policy, fix the task definition or environment bug, and evaluate adversarial cases against independent measures. Human review can help identify failure modes, but reviewers can also be fooled by the same proxy. I would not reuse the exploited reward alone as proof that the revised policy is safe or effective.
105. How do you tackle long-horizon credit assignment?
Long-horizon credit assignment asks which earlier decisions caused a delayed outcome. Sparse rewards, stochastic transitions, and many irrelevant actions make that signal noisy. I first check whether the state representation and reward timing preserve the information required to learn the dependency.
Eligibility traces and multistep returns propagate outcomes across time; actor-critic baselines reduce gradient variance; recurrent state handles relevant history; hierarchical actions can shorten the effective decision horizon. Carefully designed shaping can provide denser feedback, but arbitrary intermediate rewards may change the task.
I would use diagnostic tasks with known dependencies and compare reward delay, horizon, and success frequency. For example, if only a key collected early enables a later reward, the learner needs exploration that encounters both events and an update mechanism that connects them. Increasing network size alone does not establish either. Better credit assignment and better exploration solve related but distinct problems.
106. What are options in hierarchical reinforcement learning?
An option is a temporally extended action defined by an initiation set, an internal policy, and a termination condition. A higher-level controller selects an option, which can execute multiple primitive actions before returning control. This produces a semi-Markov decision process when durations vary.
For an option lasting k steps, a value backup uses the accumulated discounted reward during those steps plus γ^k times the next high-level value. Using γ only once would incorrectly discount an option that lasts many primitive time steps.
A navigation system might choose “reach the doorway” rather than selecting every motor command itself. Useful options can improve reuse and exploration, but poorly designed termination rules or subgoals can hide important decisions and introduce failure modes. I would evaluate both option success and overall task return. A high subgoal success rate does not establish that the high-level policy chooses the right subgoal.
107. Why does multi-agent learning create nonstationarity?
From one learner's perspective, other learning agents change the effective transition and reward distributions as their policies improve. This breaks the stationarity assumption that a single-agent learner may implicitly rely on, even if the underlying game rules remain fixed.
Replay makes this especially visible: old transitions were generated against earlier teammates or opponents. A policy may overfit those historical behaviors rather than remain effective against the current population. Exploration by several agents can also create coordination failures that resemble random environmental noise.
I would specify whether the task is cooperative, competitive, or mixed, and whether opponents are fixed during evaluation. Centralized information during training, population-based evaluation, opponent modeling, or policy fingerprints can address particular issues. There is no general guarantee of convergence for arbitrary independent deep learners in arbitrary games, so the claim must be tied to the algorithm, game assumptions, and measured evaluation protocol.
108. How does centralized training with decentralized execution work?
Centralized training with decentralized execution lets training components use information that individual agents will not have at deployment, while constraining each deployed policy to its available local observations. A centralized critic can condition on joint state and actions to reduce ambiguity when assigning credit.
For example, two robots can train with a critic that sees both positions, but each actor must choose actions using only its own sensors and any communication channel actually allowed in operation. Feeding global state into the actor during training and silently removing it later is not a valid implementation.
I would audit the information boundary explicitly and evaluate with realistic communication delays and failures. Centralized critics can become expensive as the number of agents grows, and they do not remove coordination or nonstationarity by themselves. The method is a training architecture, not proof that decentralized agents will cooperate robustly in unseen circumstances.
109. How would you formulate a recommendation problem without optimizing only clicks?
I would define the decision horizon and business objective before deciding that recommendation needs full RL. If recommendations alter future engagement, satisfaction, or available choices, sequential modeling may be justified; if effects are essentially immediate, a contextual bandit can be simpler.
Clicks are an observable signal, not the complete objective. I would consider longer-term satisfaction, retention, diversity, user controls, and adverse outcomes, with explicit constraints where trade-offs should not be left to a scalar reward. Delayed and censored outcomes need a defensible attribution scheme.
Logged recommendations are selected by an existing policy, so evaluation must address action support and selection bias. I would compare against current ranking and bandit baselines, avoid interpreting missing clicks as randomized negatives, and propose only a bounded authorized experiment. A recommendation policy that raises short-term reward while worsening user outcomes has failed the intended task, even if its RL loss looks healthy.
110. How would you design a bounded RL controller for a physical process?
I would begin with a validated baseline controller, a bounded action space, and explicit operating limits rather than allowing unrestricted exploration. The RL component might propose set-point adjustments while a separate controller handles fast stabilization and an independent layer rejects infeasible commands.
The state must include relevant delays, actuator status, and history needed for control. Rewards should reflect the true objective and costs, but emergency stops and equipment limits should not depend on the learned reward. A simulator supports development only to the extent that its dynamics and failure modes are credible.
Before any physical experiment, I would require authorization, instrumentation, a fallback path, latency checks, and stop criteria. Offline and simulation tests should include sensor failures and out-of-range inputs. This is a proposed engineering design, not a claim of safety certification or a substitute for domain-specific validation of the actual process.
111. How do you choose between RL, a contextual bandit, and conventional optimization?
I choose the simplest formulation that captures the important consequences of actions. A contextual bandit fits decisions with context-dependent immediate outcomes and no material action-induced effect on future states. Full RL is appropriate when delayed effects and sequential trade-offs are central.
Conventional optimization can be preferable when the objective, constraints, and dynamics are sufficiently known, or when decisions can be solved reliably without learning through interaction. Supervised prediction plus optimization is also a strong baseline. “The problem involves decisions” is not enough to justify RL.
I would compare data availability, experiment risk, action coverage, model fidelity, computational budget, and the value of learning long-term consequences. A fixed-horizon inventory problem with known dynamics may suit dynamic programming or mathematical optimization; unknown demand responses may motivate learning. The decision should be supported by measurable baseline gaps, not by choosing the most fashionable algorithm first.
112. How do you detect distribution shift after an RL policy is deployed?
I monitor inputs, actions, rewards, constraint outcomes, model uncertainty, and the relationship between predicted and realized value. Distribution shift can appear in sensor statistics, user populations, transition dynamics, or the policy's own induced state visitation, so an input-drift score alone is insufficient.
I would segment results by operating region and preserve a fixed reference policy or holdout where feasible. Delayed outcomes require leading indicators, but those indicators should not be confused with the final objective. An anomaly threshold should correspond to an operational response, not merely a dashboard alert.
Python example
Retraining is not an automatic remedy: shifted data may be contaminated by the failing policy. I would diagnose the change, review data provenance, and revalidate a replacement before restoring exposure.
113. What should a safe rollout and rollback plan contain?
A rollout plan specifies the eligible population, exposure limits, comparison baseline, metrics, decision rules, owners, and stopping conditions before deployment. It should begin with the least consequential stage that can resolve the relevant uncertainty, such as shadow operation or human-reviewed recommendations.
Rollback must restore more than model weights. I would version preprocessing, action mappings, reward definitions, configuration, and stateful components, and test that the fallback can run with current inputs. Replay-buffer or online-update changes may also need to be isolated so a failed policy does not silently contaminate its replacement.
I would monitor hard-limit violations separately from average reward and define who can stop the experiment immediately. Statistical decision rules should account for repeated monitoring when formal significance is claimed. A successful small rollout supports expansion only within the conditions tested; it is not evidence that every unseen segment or operating regime is safe.
114. Why must action permissions be enforced outside the learned policy?
A learned policy is an optimizer, not an authorization boundary. It may propose an invalid, unsafe, or unauthorized action because of estimation error, distribution shift, adversarial inputs, or an incorrectly specified reward. Permissions therefore need deterministic enforcement at the point of execution.
For example, a resource-allocation policy may rank actions that exceed a tenant's quota. An external service should check tenant identity, allowed operations, parameter bounds, current budget, and any required approval before executing a proposal. Rejected actions and executed actions should be logged distinctly so training data reflects what actually happened.
Action masking can prevent selection of known invalid discrete actions, but the execution layer must still handle stale state and race conditions. A mask used only during training is insufficient. The design should include a safe response when no allowed action remains rather than assuming the policy always returns a valid command.
115. How does reinforcement learning from human feedback differ from ordinary RL?
RLHF is a family of methods that use human feedback to define or learn a reward signal for reinforcement learning. Ordinary RL can use a directly specified environment reward; RLHF often substitutes a learned proxy for subjective qualities such as helpfulness or preference.
In a common language-model pipeline, supervised demonstrations initialize a policy, human comparisons train a reward model, and an RL optimizer improves the policy against that reward while limiting drift. Tokens can be actions and the prompt plus generated prefix the state, although some analyses treat the complete response as a single action.
RLHF is not synonymous with all language-model training, all preference learning, or a deployed chatbot taking actions. Human feedback can also train non-language control systems. Because the learned reward is only a proxy, improvements in reward score do not by themselves prove truthfulness, safety, or agreement with every user's values.
116. What is the standard preference-based reward-model training objective?
A common pairwise reward model uses a Bradley–Terry-style likelihood: P(y_w preferred to y_l | x)=σ(rφ(x,y_w)−rφ(x,y_l)). Training minimizes the negative log likelihood of the preferred response. The model learns relative scores, not an externally calibrated unit of human value.
If the score difference is 2, the modeled preference probability is about 0.881. Adding the same constant to both scores leaves this probability unchanged, which illustrates why pairwise data does not identify an absolute reward offset.
Python example
I would evaluate held-out comparisons, disagreement across annotators, prompt coverage, and sensitivity to superficial cues such as length. The probabilistic preference model is an assumption; real preferences can be inconsistent, context-dependent, and multidimensional.
117. How does PPO-based RLHF balance reward improvement and reference-policy drift?
A common RLHF objective maximizes expected learned reward minus β times KL divergence from a reference policy. The reference is often a supervised fine-tuned model. The reward encourages preferred responses, while the KL term discourages excessive distributional drift.
For a sampled response, the log ratio log πθ(y|x)−log πref(y|x) is a Monte Carlo contribution whose expectation under πθ equals KL(πθ||πref). An individual sampled log ratio can be negative even though the expected KL is nonnegative. Token-level implementations sum appropriate log probabilities over the response.
PPO's clipped update objective is a separate mechanism that limits certain policy-update incentives relative to the rollout policy; it is not identical to the reference-policy KL penalty. I would inspect both the optimization configuration and actual drift, reward, and independent quality metrics. Neither clipping nor a finite KL coefficient guarantees semantic safety or prevents reward exploitation.
118. How does Direct Preference Optimization differ from PPO-based RLHF?
Direct Preference Optimization trains directly on preference pairs using policy and reference-policy log-probability ratios. It derives a classification-style loss from a KL-regularized reward-maximization formulation, avoiding a separately fitted explicit reward model and online RL rollouts in the standard offline DPO procedure.
For winner y_w and loser y_l, define z = β[(log πθ(y_w|x)−log πref(y_w|x))−(log πθ(y_l|x)−log πref(y_l|x))]. The loss is −log σ(z). Increasing the winner's reference-relative log probability compared with the loser's decreases this loss.
This does not mean DPO is unrelated to RL objectives, nor that every preference-training method is PPO-based RLHF. Its derivation depends on the reward/preference model and regularized objective, while practical success depends on data coverage and optimization. I would compare DPO and PPO using independent task evaluations rather than claiming one dominates across all datasets or deployment requirements.
119. How can preference optimization overfit or exploit a learned reward?
Preference optimization can exploit errors in the reward model or preference dataset instead of improving the intended behavior. A policy may learn superficial cues such as verbosity, confident wording, or familiar formatting if those cues correlate with positive labels. Stronger optimization can magnify these shortcuts.
I would track independent human evaluations, factual or task-specific checks, subgroup behavior, and adversarial examples alongside training reward. Reward-model accuracy on a static held-out set can deteriorate as the policy generates a new distribution. DPO also inherits dataset biases and coverage limits even though it lacks a separately trained explicit reward model.
Mitigations include diverse comparisons, clear annotation criteria, uncertainty-aware review, held-out prompt families, controlled optimization strength, and fresh evaluation of policy outputs. No single mitigation proves alignment. If reward rises while independently judged quality falls, I would treat that as evidence to stop and investigate rather than celebrate the optimization curve.
120. How would you review an RL system before approving a production experiment?
I would review whether the experiment has a defensible objective, relevant baselines, valid data, reproducible implementation, and a bounded authorization and safety plan. The key question is whether the proposed exposure is justified by evidence, not whether the training curve looks impressive.
My technical review would check reward and constraint definitions, state sufficiency, action support, termination handling, evaluation leakage, uncertainty, and performance across important segments. I would require documented limits for any offline evaluator or simulator and distinguish measured results from assumptions and illustrative tests.
Operationally, I would verify external permission checks, observability, fallback readiness, rollback, ownership, and stop conditions. A small local numerical fixture can validate a formula or indexing convention; it does not establish convergence, policy quality, real-world safety, or production reliability. Approval should name the permitted experiment and its bounds explicitly, leaving broader deployment contingent on the evidence obtained.
Reinforcement Learning Documentation and Further Reading
This guide emphasizes durable concepts and scoped examples, not an algorithm ranking. Library APIs and environment wrappers evolve; verify installed versions and task definitions. Local arithmetic and algorithm fixtures do not establish learning convergence, policy quality, operational safety, or production performance.
These primary research papers, the foundational textbook, and official environment documentation provide a curated route into the topics covered. Published benchmark results apply to their experimental settings rather than establishing universal algorithm superiority.
Sutton and Barto — Reinforcement Learning An Introduction, second edition
Mnih et al. — Playing Atari with Deep Reinforcement Learning
Fujimoto et al. — Addressing Function Approximation Error in Actor-Critic Methods
Kumar et al. — Conservative Q-Learning for Offline Reinforcement Learning
Kostrikov et al. — Offline Reinforcement Learning with Implicit Q-Learning
Agarwal et al. — Deep Reinforcement Learning at the Edge of the Statistical Precipice
Christiano et al. — Deep Reinforcement Learning from Human Preferences
Ouyang et al. — Training Language Models to Follow Instructions with Human Feedback
Janner et al. — When to Trust Your Model: Model-Based Policy Optimization
Similar Posts