Introduction

Reward shaping is a fundamental component of reinforcement learning, and KL penalties commonly appear when training AI agents to remain close to a reference policy. A widespread misconception arises when a sampled log-probability ratio turns out negative: the immediate instinct is to clamp it to zero, assuming the calculation must be flawed. This article unpacks why a negative value is often expected, what it actually represents, and how to handle it without disrupting your training pipeline.

What Happened

When an AI code agent generates a response, practitioners frequently compare the token log probabilities under the current policy against those from a fixed reference policy. The per-token log ratio can be negative for individual outcomes, even though the exact forward KL divergence between two normalized distributions is guaranteed to be non-negative. The article demonstrates this with a simple two-outcome Python calculation, showing that a single sampled ratio below zero does not indicate a systemic error—it is simply a natural consequence of sampling from a distribution that assigns lower probability to that outcome.

Why This Matters

Clipping the penalty to max(0, penalty) may seem like a harmless numerical fix, but it changes the estimator you are optimizing. The piece walks through exact forward KL, clipped expectation, and why the mean of sampled log ratios differs from a sum, especially when token counts vary. It also references the RLEF framework, which uses geometric-mean response probabilities rather than raw token products, and emphasizes that reward bookkeeping should track the base reward, sampled mean log ratio, coefficient, and final shaped reward as separate components.

Key Takeaways

  • A negative sampled log ratio is expected and does not by itself invalidate the reward calculation.
  • Exact forward KL divergence is always non-negative, but individual samples can be negative.
  • Clipping the penalty alters the objective and should be a deliberate design choice, not a default repair.
  • Mean and sum of log ratios produce different results when token lengths vary; always verify which your objective expects.
  • Log reward components separately—base reward, mean log ratio, coefficient, and final shaped reward—to make diagnostics transparent.
  • If using RLEF or PPO-style KL regularization, confirm whether your reference policy is fixed or evolving, and whether your estimator uses mean or sum normalization.

Conclusion

Before "fixing" a negative KL penalty, identify whether you are working with a distribution-level divergence, a single sampled log ratio, or a length-normalized response statistic. Each carries different guarantees, and mistaking one for the other can lead to incorrect objective modifications. By keeping diagnostic components visible and understanding the math behind the penalty, you can shape rewards confidently and keep your agent's training trajectory on track.