MedMCQA

4-choice medical-board questions. The model outputs a verbalized probability distribution over the choices. Replication of Bereket & Leskovec, Uncalibrated Reasoning: GRPO Induces Overconfidence for Stochastic Outcomes.

GRPO divides the group-centered reward by the within-group reward standard deviation $\hat\sigma_i$,

Dropping the std leaves the advantage unbiased up to $\tfrac{G-1}{G}$ → calibrated. Keeping it, the std is taken per outcome — for a binary answer, $\sigma_1$ when $a{=}1$ and $\sigma_0$ when $a{=}0$. As the policy concentrates above $0.5$, $\sigma_0 > \sigma_1$, so the penalty for the over-confident wrong call is divided by the larger number and shrinks → overconfidence.

This is not necessarily true in general — examined further on GRPO σ-normalization.

Dataset entry

Which of the following is not true for myelinated nerve fibers:

A. Impulse through myelinated fibers is slower than non-myelinated fibers
B. Membrane currents are generated at nodes of Ranvier
C. Saltatory conduction of impulses is seen
D. Local anesthesia is effective only when the nerve is not covered by myelin sheath

{A: 0.4; B: 0.3; C: 0.1; D: 0.2}

Setup

Qwen3-4B + LoRA, GRPO. Log-score reward on the verbalized distribution, trained with vs without normalizing by the group reward std.

Results

Eval on 4183 questions:

The ECE (expected calibration error) validates their result. wandb report.