← All writing
Reinforcement LearningReward Design

A coin flip made this model better at math. Or did it?

Notes on "Spurious Rewards", a paper about what reward-based training really teaches a model

Dhruvi Paprunia3 min read
Coin flip reward and math performance illustration

Tap the image to enlarge.

Reinforcement learning (RL) is supposed to work because of the reward. You tell the model what a good answer looks like, and it learns to give more of them. Get the reward wrong and the model should learn the wrong thing.

In "Spurious Rewards: Rethinking Training Signals in RLVR", Sewon Min and her co-authors put that idea to the test. They trained a math model with rewards that made no sense. Some were random. Some rewarded only wrong answers. The model still got much better at math.

I spent an evening with this paper, and the reason it works is more interesting than the result itself.

First, what is RLVR?

RLVR stands for reinforcement learning with verifiable rewards. You give a model a question with a checkable answer, like a math problem. The model tries a few answers, a program checks each one, and correct answers earn a reward of 1. Wrong answers get 0.

One way to train this way is GRPO, short for Group Relative Policy Optimization. It compares several answers to the same question and nudges the model toward answers whose rewards were above the group average.

The idea is simple: the reward tells the model what "good" looks like.

The surprising result

The team took Qwen2.5-Math-7B, a model already trained heavily on math, and swapped the real reward for worse and worse ones:

The researchers tested the model on MATH-500, a set of math problems used to measure accuracy. Training with rewards for correct answers raised accuracy by 29.1 percentage points. Random rewards raised it by 21.4 percentage points, meaning 21.4 more correct answers per 100 questions. A coin flip got most of the way there.

Ground-truth and random reward gains on MATH-500

Tap the image to enlarge.

So the reward alone cannot explain the improvement. What else in the training could be doing the work?

Why would a coin flip help?

But that reward comparison is not the whole training rule. GRPO also uses clipping.

Clipping is one rule inside that training. Think of it as saying, 'You have already changed enough in that direction. This part of the score gives no extra credit for pushing further.' It limits the incentive, not the change itself.

Here is a toy example. A token is a small piece of text, sometimes a word and sometimes part of one.

Suppose the model gave a token a 10% chance when it generated the training answers. With a 20% clipping setting, the upper threshold is 12%, not 30%.

If feedback favors making that token more likely, pushing past 12% earns no extra credit from this part of the training score. Its probability can still change through other training effects. Clipping is not a hard wall.

Illustrative clipping example: a token starts at 10%, with an upper threshold of 12%. Extra credit stops beyond that threshold for positive feedback, but probability is not capped.

Tap the image to enlarge.

Why use it? A few rewarded answers should not push a model so far that it loses useful behavior it already had. Clipping discourages those large changes, helping training take more measured steps. But it cannot tell whether a large change would be a mistake or genuine progress.

That last tradeoff is important in this paper. The authors find that clipping can bias training toward behavior the model already tends to use, even when rewards are random. In their experiments, removing clipping stopped random rewards from producing consistent gains. This does not mean every no-clipping run gets worse, or that clipping alone explains every gain.

That gives us a possible link: training can strengthen behavior the model already uses. The next question is whether that behavior helps it solve math.

The useful habit Qwen already had

For Qwen2.5-Math, one of its habits happens to be very good. It often "reasons in code", writing Python-style steps even though nothing runs the code. Answers written this way were right 60.9% of the time, compared to 28.0% without.

Before training, 65% of its answers used code reasoning. After training with misleading rewards, such as random rewards or rewards for wrong answers, it was over 90%. The paper calls these spurious rewards.

This suggests that much of the gain came from using an existing useful habit more often, rather than learning a new math skill from random feedback. Code reasoning is one example, not a complete explanation of every gain.

Code reasoning frequency and accuracy

Tap the image to enlarge.

What this doesn't mean

It doesn't mean a random reward will make any model better. If training turns up a common habit, it can turn up a bad one too. The team tested Llama and OLMo, two other model families, as well; their gains with misleading rewards were mostly flat and sometimes negative. Qwen2.5-Math had a useful code-reasoning habit to begin with. The coin flip didn't create it.

And it doesn't mean the reward method deserves all the credit. Qwen is a common starting point for RLVR research, so a result that looks strong on Qwen may be leaning on the model's existing strengths. That's why the paper tests other model families and calls for future work to do the same.

Two controls I would try in tool use

These are proposed experiments, not results from this paper or my own run. First, I would compare my correct reward with randomly assigned rewards using the same starting model, data and training budget. Multiple seeds and a base model with no extra training would show how much each comparison varies.

To check the clipping effect, I would run correct and random rewards across the same clip ranges while keeping the other training choices fixed. I would compare exact call accuracy and how often clipping changes the updates across several seeds. A changed score alone would not prove the mechanism, so I would also track the share of updates affected by clipping and the model's probability changes.

I would keep validation prompts separate from the final unseen test prompts. These controls check whether the tool use setting behaves like the math setting. They do not assume that the paper's Qwen math result transfers to every model or task.

What I'm taking from this

  1. Check RL results on more than one model family. The paper makes this point strongly.
  2. Use random rewards as a comparison. If they produce much of the same gain, the reward's accuracy may not explain all of the improvement.
  3. Know what the model already does well before training. RL often turns up existing skills more than it teaches new ones.

Having written reward functions myself, I used to think of the reward as the whole story. This paper made me look at the model first.


Read the paper: Spurious Rewards: Rethinking Training Signals in RLVR

Dhruvi Paprunia3 min read

Keep reading

Continue with concrete tool-use training, rare correct answers, and the choice of which problems get training effort.

Related readThe polite sentence that broke my modelRelated readYour model found the right answer. It might never learn to find it again.Related readNever Give Up: why easy problems eat the RL budget

All writing →