close
License: CC BY 4.0
arXiv:2509.03403v2 [cs.LG] 15 May 2026

Beyond Correctness: Harmonizing Process and Outcome Rewards through RL Training

Chenlu Ye1,2  Zhou Yu1  Ziji Zhang1  Hao Chen1  Narayanan Sadagopan1 Email: chenluy3@illinois.edu    Jing Huang1  Tong Zhang2  Anurag Beniwal1
1Amazon   2University of Illinois Urbana-Champaign
Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) improves final-answer accuracy on reasoning tasks, but it does not reliably improve reasoning quality. Because outcome rewards only assess final answers, they also reward spurious successes: flawed reasoning can still receive maximal reward when it accidentally reaches the correct outcome. This outcome reward hacking creates biased gradients, making current RLVR insufficient for learning faithful reasoning. Process Reward Models (PRMs) provide step-wise supervision, but directly optimizing PRMs or naively combining them with outcome rewards is unstable under distribution shift during RL training process. We introduce PRocess cOnsistency Filter (PROF), a data curation method that uses PRM–ORM consistency for sample selection rather than direct reward optimization. PROF keeps correct responses with strong process support and incorrect responses with weak process support while maintaining a balanced training ratio. Experiments show that PROF consistently improves both final-answer accuracy and intermediate reasoning quality over strong baselines, with less dependence on strong PRMs. Codes and training recipes are available at https://github.com/amazon-science/PROF-GRPO.

Beyond Correctness: Harmonizing Process and Outcome Rewards through RL Training

Chenlu Ye1,2††thanks: Email: chenluy3@illinois.edu   Zhou Yu1   Ziji Zhang1   Hao Chen1   Narayanan Sadagopan1

Jing Huang1   Tong Zhang2   Anurag Beniwal1 1Amazon   2University of Illinois Urbana-Champaign

Problem: There are 44 coins of 1,2,3,51,2,3,5 cruzeiros, which weigh 1,2,3,51,2,3,5 grams respectively. One of them is counterfeit, differing in weight (but not known whether it is heavier or lighter). How can the counterfeit coin be identified with the minimum number of weighings using a balance scale without weights?  Step 1: Introduction. We need to identify a counterfeit coin with an unknown weight difference from … Step 2 - 6: The Flawed Weighing Logic and Code.
1. Weigh coins of 1 gram and 2 grams against coins of 3 grams and 5 grams. 2. Based on the result of the first weighing: - If the left side is heavier, then the counterfeit coin is either 1 gram or 3 grams. - If the right side is heavier, then the counterfeit coin is either 2 gram or 5 grams. - If both sides are equal, then the counterfeit coin is either 2 gram or 5 grams. 3. For the second weighing, … Let’s illustrate this with code and ensure that the steps are correct … output: (1, ’heavier’). Step 7, 8, 9: Summary and Final Answer.
Here is a summary of the steps: 1. Weigh coins of 1 gram and 2 grams against coins of 3 grams and 5 grams… 2. For the second weighing, weigh the 1 gram coin against a known genuine coin (3 grams)… Thus, the minimum number of weighings required to identify the counterfeit coin is 2\boxed{2}.
Table 1: An example of reaching a correct final answer through flawed reasoning. The first weighing ({1g,2g} vs. {3g,5g}) is invalid because the two sides are not equal in nominal weight, so all subsequent deductions are unsupported even though the final answer happens to be correct.

1 Introduction

Verifiable rewards have attracted substantial attention because they can reliably improve performance on reasoning tasks with easily verifiable outcomes, such as mathematical and coding problems (Cobbe et al., 2021; Jaech et al., 2024; Shao et al., 2024; Xiong et al., 2025b). However, success on these tasks is usually measured only by the final answer, while in many applications we also care about the quality of the reasoning process itself, especially its faithfulness, validity, and interpretability. Throughout this paper, we use reasoning quality as an umbrella term for these process-level properties. Optimizing the verifier is therefore not the same as optimizing reasoning quality. Because verifiers only assess final outcomes, Outcome Reward Models (ORMs) are too sparse and coarse to distinguish flawed reasoning within correct answers or valid reasoning within incorrect answers. For instance, the training example in Table 1 has fundamentally invalid reasoning but still arrives at the correct answer. To theoretically analyze this challenge, we define a latent state variable zz, where z=1z=1 denotes a valid intermediate reasoning process (i.e., no error). Let απ=ℙ​(z=1|π)\alpha_{\pi}=\mathbb{P}(z=1|\pi) represent the probability of generating valid reasoning. Given that an incorrect process (z=0z=0) may coincidentally yield a correct answer (r=1r=1) with a small probability ϵ\epsilon (i.e., ℙ​(r=1|z=0)≤ϵ\mathbb{P}(r=1|z=0)\leq\epsilon), the expected reward can be decomposed as:

𝔼π​[r]\displaystyle\mathbb{E}_{\pi}[r] =∑z∈0,1P​(r=1|z)​P​(z|π)≈απ+ϵ​(1−απ)\displaystyle=\sum_{z\in{0,1}}P(r=1|z)P(z|\pi)\approx\alpha_{\pi}+\epsilon(1-\alpha_{\pi})
=(1−ϵ)​απ+ϵ.\displaystyle=(1-\epsilon)\alpha_{\pi}+\epsilon.

While the ideal objective is to maximize απ\alpha_{\pi}, the term ϵ​(1−απ)\epsilon(1-\alpha_{\pi}) introduces gains from spurious successes. During training, samples where r=1r=1 despite z=0z=0 generate biased gradients that inadvertently reinforce flawed reasoning paths, allowing the policy to increase outcome reward without improving the latent reasoning quality. This creates a process-outcome mismatch: final-answer correctness no longer reliably reflects reasoning quality, especially the faithfulness of the underlying reasoning process. We refer to the resulting optimization failure as outcome reward hacking: the model is rewarded for exploiting weaknesses in outcome-only supervision rather than for producing faithful reasoning. This misalignment leads to unfaithful reasoning, a limitation increasingly observed in recent studies (Baker et al., 2025; Chen et al., 2025b). Consequently, relying solely on final answer accuracy is insufficient; ensuring reasoning quality, faithfulness, and interpretability in Chain of Thought (CoT) is crucial for the safety and practical utility of LLMs (Zhu et al., 2025; Lyu et al., 2023; Yeo et al., 2024). To empirically support this process-outcome mismatch and the resulting reasoning-quality gap, we later analyze 2k samples from Qwen2.5-Math-7B and find that 26.28%26.28\% of correct responses still contain flawed reasoning, as judged by Claude. Within this flawed-correct subset, PROF identifies and filters 65.88%65.88\% (Figure 1).

Refer to caption
Figure 1: Within correct responses that contain flawed reasoning, PROF filters 65.88%65.88\% and leaves 34.12%34.12\%.

This process-outcome mismatch shows that current RLVR alone cannot solve the reasoning-quality gap: outcome rewards are necessary for verifiability, but insufficient for supervising how the answer is reached. This has motivated a flurry of recent work on training Process Reward Models (PRMs) and using them in RL training (Lightman et al., 2023; Zhang et al., 2025; Zou et al., 2025), since PRMs provide dense and fine-grained feedback over intermediate reasoning processes. In other words, if we want to optimize reasoning quality rather than final correctness alone, some form of PRM-style process supervision is necessary. However, directly using PRMs as rewards introduces a second failure mode. Although these PRMs achieve excellent performance on PRM benchmarks, directly combining PRM and ORM in the reward function can lead to reward hacking. Notably, since PRMs are often trained offline, applying them to online training suffers from distribution shift. Especially in boundary cases where the policy encounters difficult problems and produces rarely seen responses, PRMs often fail to judge them correctly, thus leading to severe reward hacking when they are used as explicit reward signals during RL training (Michaud et al., 2020; Tien et al., 2022). Even when some works (Zha et al., 2025; Cui et al., 2025) attempt to co-train the policy and PRMs online, they can only train PRMs in implicit ways that lack accurate process scores, such as implicit generative rewards or alignment between process rewards and outcomes. Therefore, instead of training another PRM for a specific dataset or base model, we focus on how to robustly integrate a pre-trained PRM into online training, i.e., how to harmonize accurate but coarse-grained ORMs with fine-grained but noisy Process Reward Models (PRMs) in Reinforcement Learning (RL).

We develop a PRocess cOnsistency Filter (PROF) framework, an online data curation strategy based on process-outcome consistency. PROF oversamples responses at training time and then ranks and filters them by PRM–ORM consistency. Specifically, it removes samples where the process and outcome signals conflict, such as correct responses derived from flawed reasoning or incorrect responses that contain sound reasoning steps. By using PRMs for filtering rather than as direct optimization targets, PROF injects process supervision into RLVR while avoiding the instability of explicit PRM reward maximization. Furthermore, because correct and incorrect responses have different consistency distributions, we rank each group separately to maintain a balanced training ratio. PROF is a modular framework that can be combined with RL algorithms like Group Relative Policy Optimization (GRPO) for online training.

Refer to caption
Figure 2: Visualization of Algorithm 1. Rectangle length denotes trajectory-level process score. PROF ranks correct and incorrect groups separately by PRM–ORM consistency: in the correct group, higher-consistency samples are kept; in the incorrect group, lower-consistency (or random) samples are kept to maintain a balanced correct/incorrect ratio.

We conduct extensive experiments to validate the improvement of PROF on both outcome accuracy and reasoning quality using both Qwen (Yang et al., 2024) and LLaMA (Dubey et al., 2024) models. To summarize, we highlight our key contributions as follows:

  • •

    We identify a fundamental reasoning-quality gap in current RLVR. Because outcome-only rewards can reward spurious successes, current RLVR can improve final-answer accuracy without reliably improving faithful reasoning, a failure mode we characterize as outcome reward hacking. We support this process-outcome mismatch with both theoretical analysis and empirical evidence.

  • •

    We propose PROF, a consistency-based data curation framework that robustly injects PRM supervision into RLVR. Rather than directly optimizing PRM scores or naively blending PRM and ORM rewards, PROF uses PRM–ORM consistency for ranking and filtering, allowing it to remove conflicting trajectories while maintaining a balanced correct/incorrect training ratio.

  • •

    Extensive experiments and ablations on both Qwen and LLaMA models show that PROF consistently improves both final-answer accuracy and intermediate reasoning quality over strong baselines, with less dependence on strong PRMs. Under matched compute cost and matched rollout-group size, PROF still achieves larger gains by almost 2%2\%. We further demonstrate robustness to different off-the-shelf PRMs, generality beyond GRPO, and the importance of filtering correct and incorrect responses separately.

2 Related Work

Reasoning-Quality Gaps and Faithfulness of Chain-of-Thought.

A growing literature documents process-outcome mismatch in language models: final-answer correctness can diverge substantially from reasoning quality, especially from the faithfulness of a model’s verbalized reasoning. Turpin et al. (2023) show that CoT explanations can omit biasing features and rationalize incorrect predictions, while Lyu et al. (2023) argue that standard CoT does not guarantee a faithful explanation of how the answer is produced. Subsequent work measures this reasoning-quality gap more directly: Nguyen et al. (2024) report a significant disparity between answer accuracy and CoT faithfulness in multi-hop question answering, and Paul et al. (2024) use causal mediation analysis to show that LLMs do not reliably use their generated intermediate steps when producing the final answer. Beyond faithfulness, Yeo et al. (2024) advocate evaluating reasoning explanations along multiple axes including robustness and utility, and Jacovi et al. (2024) show that even dedicated verifiers struggle to detect logical errors and contradictions inside reasoning chains. Recent monitoring work extends these concerns to reasoning models themselves: Baker et al. (2025); Chen et al. (2025b) show that model-generated reasoning often fails to transparently reveal the cues or considerations that drive behavior. Relative to this line of work, we focus on the RLVR setting and study how process supervision can reduce process-outcome mismatch during online training by filtering trajectories whose final outcomes and reasoning quality are inconsistent.

Process-Supervised Reward Models for Fine-Grained Feedback.

RLHF focuses on trajectory-level comparison under the Bradley-Terry model. For reasoning-related tasks, Yang et al. (2024) uses final-answer correctness to construct preference pairs and trains Bradley-Terry reward models for mathematical reasoning. A more widely used approach, termed Outcome Reward Models (ORMs), trains a classifier to predict whether the final answer is correct based on the reasoning history. However, Lightman et al. (2023) show that Process-Supervised Reward Models (PRMs), which evaluate each intermediate step of a reasoning chain, significantly outperform ORMs, especially for data selection tasks such as best-of-n sampling (Lightman et al., 2023). Their approach, however, requires human annotators to label each intermediate step. Wang et al. (2023) proposes using Monte-Carlo estimation of the Q value to determine labels automatically. Many follow-up works improve PRMs through generative reward modeling, advanced training techniques such as RL, and refined engineering practices (Xiong et al., 2024b; Zhang et al., 2025; Khalifa et al., 2025; Zhao et al., 2025; Xiong et al., 2025c). Our work does not focus on improving PRMs themselves; instead, we use PRMs to supervise the intermediate steps of CoT trajectories for data filtering. We mainly use Qwen2.5-Math-PRM-7B from Zhang et al. (2025) because it is trained on the Qwen distribution and achieves strong performance on ProcessBench (Zheng et al., 2024).

Sample Filtering in Reinforcement Learning for LLM.

A key challenge in applying reinforcement learning to LLM applications is the imperfection of reward signals. These signals stem from a learned reward model, such as Reinforcement Learning from Human Feedback (RLHF), or are sparse, delivered only at the end of a trajectory (e.g. RLVR). In RLHF, the reward model is trained on human-annotated pairwise comparisons, typically using a Bradley-Terry model (Bradley and Terry, 1952). Due to inherent human disagreement and finite training data, the model develops shortcuts that RL algorithms can exploit (Lin et al., 2023; Eisenstein et al., 2023) to chase for a fake high reward. Consequently, these rewards may not fully align with the underlying intended goals, leading to reward hacking.

Data filtering has proven effective in mitigating reward hacking across RL-based LLM training. In RLHF, prior work filters preference pairs by reward gap (Yuan et al., 2024; Dong et al., 2024; Xiong et al., 2024a; Zhang et al., 2024) or combines reward with response length (Kim et al., 2024; Yu et al., 2025a) to retain samples that are more reliable under the learned reward model.

Filtering is also useful in RLVR despite the reward being available only at the final outcome. Rejection sampling fine-tuning discards incorrect trajectories and often approaches stronger RL baselines (Dong et al., 2023; Chen et al., 2025a; Xiong et al., 2025a). Other methods filter prompts by difficulty (Yang et al., 2024), remove zero-gradient prompts via dynamic sampling (Yu et al., 2025b), or over-sample and retain subsets that improve reward variance or the balance between correct and incorrect responses (Xiong et al., 2025a; Xu et al., 2025). In contrast to these methods, which mainly rely on coarse outcome-level signals, our approach uses process-supervised reward models (PRMs) (Lightman et al., 2023) to filter trajectories based on reasoning quality at the level of intermediate steps and their consistency with ORMs.

3 Formulation and Algorithm

An LLM defines a policy distribution: given a prompt xx, it assigns density π​(a|x)\pi(a|x) to each response aa. For mathematical reasoning tasks with a binary verifiable reward, there exists a verifier mapping prompt-response pairs (x,a)(x,a) to a scalar reward ro​(x,a)∈{−1,1}r^{o}(x,a)\in\{-1,1\}. For each prompt, we generate a group of responses together with their verifier outcomes, denoted by {(ai,rio)}i=1G\{(a_{i},r^{o}_{i})\}_{i=1}^{G}.

Algorithm 1 Process Consistency Filter (PROF)
1: Input: Number of rollouts nn, policy update size mm, rollouts {a1,…,an}\{a_{1},\ldots,a_{n}\}, outcome rewards {r1o,…,rno}\{r^{o}_{1},\ldots,r^{o}_{n}\}, step number regularization parameter λ,Hλ>0\lambda,H_{\lambda}>0.
2: Obtain process rewards for each rollout aia_{i} with HiH_{i} steps: (ri1,…,riHi)(r^{1}_{i},\ldots,r^{H_{i}}_{i}) and compute trajectory-wise consistency
ripro=[1Hi​∑h=1Hirih−λ​I​(Hi=1​ or ​Hi≥Hλ)]⋅rio.\displaystyle r^{\mathrm{pro}}_{i}=\Big[\frac{1}{H_{i}}\sum_{h=1}^{H_{i}}r^{h}_{i}-\lambda I(H_{i}=1\text{~or~}H_{i}\geq H_{\lambda})\Big]\cdot r^{o}_{i}. (1)
3: Divide rollouts into correct group 𝒢+={a1+,…,an++}\mathcal{G}_{+}=\{a^{+}_{1},\ldots,a^{+}_{n_{+}}\} with rio=1r^{o}_{i}=1 and incorrect group 𝒢−={a1−,…,an−−}\mathcal{G}_{-}=\{a^{-}_{1},\ldots,a^{-}_{n_{-}}\} with rio=−1r^{o}_{i}=-1, where n++n−=nn_{+}+n_{-}=n.
4: Compute kept number k+∈[n+],k−∈[n−]k_{+}\in[n_{+}],k_{-}\in[n_{-}] in each group such that k++k−=mk_{+}+k_{-}=m and k+​k−k_{+}k_{-} is maximized.
5: Rank 𝒢+\mathcal{G}_{+} and 𝒢−\mathcal{G}_{-} by rpror^{\mathrm{pro}} separately, and for the correct group keep the samples 𝒦+={ai+|rank​(rpro​(ai+))≤k+}\mathcal{K}^{+}=\{a^{+}_{i}|\mathrm{rank}(r^{\mathrm{pro}}(a^{+}_{i}))\leq k_{+}\}.For the incorrect group, we have two sub-algorithms:
  • •

    PROF-POS: randomly pick k−k_{-} samples from 𝒢−\mathcal{G}_{-};

  • •

    PROF-BOTH: keep 𝒦−={ai−|rank​(rpro​(ai−))≥n−−k−}\mathcal{K}^{-}=\{a^{-}_{i}|\mathrm{rank}(r^{\mathrm{pro}}(a^{-}_{i}))\geq n_{-}-k_{-}\}.

6: Output: The kept trajectories 𝒦+∪𝒦−\mathcal{K}^{+}\cup\mathcal{K}^{-} with final kept size mm.

GRPO.

(Shao et al., 2024) proposes this policy gradient algorithm that simplifies the Proximal Policy Optimization (PPO) (Schulman et al., 2017) by only computing the advantage based on the outcome rewards in a group. Instead of maintaining and updating another value network, GRPO computes the advantage by standardizing the outcome rewards within a group: for i=1,…,ni=1,\ldots,n,

Ai=ro​(x,ai)−mean​({ro​(x,aj)}j=1n)std​({ro​(x,aj)}j=1n)+δ,\displaystyle A_{i}=\frac{r^{o}(x,a_{i})-\text{mean}\left(\{r^{o}(x,a_{j})\}_{j=1}^{n}\right)}{\text{std}\left(\{r^{o}(x,a_{j})\}_{j=1}^{n}\right)+\delta},

where ro​(x,ai)r^{o}(x,a_{i}) is the outcome reward for a given response and δ>0\delta>0 is a small constant for numerical stability. Let aita_{i}^{t} denote the tt-th token and ai<ta_{i}^{<t} denote (ai1,…,ait−1)(a_{i}^{1},\ldots,a_{i}^{t-1}). This advantage is then incorporated into a clipped surrogate objective, which is optimized to update the policy from πθold\pi_{\theta_{\text{old}}} to πθ\pi_{\theta}:

𝒥GRPO​(θ)=\displaystyle\mathcal{J}_{\text{GRPO}}(\theta)= 𝔼x∼𝒟[1n∑i=1n∑t=1|ai|min(πθ​(ait|x,ai<t)πθold​(ait|x,ai<t)Ai,\displaystyle\mathbb{E}_{x\sim\mathcal{D}}\bigg[\frac{1}{n}\sum_{i=1}^{n}\sum_{t=1}^{|a_{i}|}\min\Big(\frac{\pi_{\theta}(a_{i}^{t}|x,a_{i}^{<t})}{\pi_{\theta_{\text{old}}}(a_{i}^{t}|x,a_{i}^{<t})}A_{i},
clip(πθ​(ait|x,ai<t)πθold​(ait|x,ai<t),1−ϵ,1+ϵ)Ai)].\displaystyle\quad\text{clip}\left(\frac{\pi_{\theta}(a_{i}^{t}|x,a_{i}^{<t})}{\pi_{\theta_{\text{old}}}(a_{i}^{t}|x,a_{i}^{<t})},1-\epsilon,1+\epsilon\right)A_{i}\Big)\bigg].

Although this approach stabilizes the online policy optimization and is efficient, the sparse reward signal limits further improvement in intermediate reasoning quality.

Process Reward Model (PRM).

For a response aa composed of multiple reasoning steps a=(a1,…,aH)a=(a^{1},\ldots,a^{H}), we follow previous works (Zheng et al., 2024; Zhang et al., 2025; Zou et al., 2025) to use a newline as a sign for a new step. For each step aha^{h}, the PRM score rhr^{h} maps it, the previous steps and the prompt (x,a≤h)(x,a^{\leq h}) to a scalar rh​(x,a≤h)r^{h}(x,a^{\leq h}), where we use the short-hand notation a≤h=(a1,…,ah)a^{\leq h}=(a^{1},\ldots,a^{h}).

Our Method PROF: Process Consistency Filter Framework

We propose PROF in Algorithm 1 to robustly incorporate PRM–ORM consistency after the rollout phase, and also visualize it in Figure 2. First, we generate nn samples and obtain outcome rewards. Then, we call the PRM to generate step-level rewards for each rollout and compute the trajectory-wise consistency score rpror^{\mathrm{pro}} by taking the mean over step-level rewards and adding a step-length regularization in equation 1, where λ\lambda is the regularization parameter and HλH_{\lambda} is the threshold for the penalized step number. This regularization ensures that samples with no step segments or over-long steps are discarded in the correct group. The samples are divided into two subgroups: 𝒢+\mathcal{G}_{+} contains correct samples with ro=1r^{o}=1, and 𝒢−\mathcal{G}_{-} contains incorrect samples with ro=−1r^{o}=-1. Inspired by (Xu et al., 2025), the numbers to keep in each subgroup, k+k_{+} and k−k_{-}, are chosen to maximize the outcome-reward variance of the final kept samples k+​k−/(k++k−)2k_{+}k_{-}/(k_{+}+k_{-})^{2}. Since k++k−=mk_{+}+k_{-}=m is fixed, k+​k−=k+​(m−k+)k_{+}k_{-}=k_{+}(m-k_{+}) should be maximized, and the maximum is attained when k+k_{+} is closest to m/2m/2 under the constraint k+≤n+,k−≤n−k_{+}\leq n_{+},k_{-}\leq n_{-}. This implies that the ratio of correct and incorrect responses should be balanced. After that, for the correct group, we use rpror^{\mathrm{pro}} to rank and keep the top k+k_{+} samples. For the incorrect group, PROF-POS randomly filters samples, while PROF-BOTH uses rpror^{\mathrm{pro}} to rank and keep the bottom k−k_{-} samples. Finally, we collect the kept mm trajectories for policy update.

False Positives Are Frequent and Filterable.

We provide empirical evidence in Figure 1 to justify the practical motivation. On 2k samples from Qwen2.5-Math-7B, we find that 26.28%26.28\% of correct responses still exhibit flawed reasoning, as judged by Claude. Crucially, within this flawed-correct subset, when PROF filters the bottom half of correct responses by PRM consistency, it identifies and removes 65.88%65.88\% of these flawed responses. This confirms that process-outcome mismatch is a critical bottleneck and that PROF effectively filters problematic samples to improve gradient quality.

4 Experiments

4.1 Setup

We focus on mathematical reasoning tasks in this work. For online training, we use the Numina-Math prompt set (Beeching et al., 2024), which contains nearly 860k math problems with ground-truth answers ranging from Chinese high school exercises to US and international mathematics olympiad problems. We use Qwen2.5-Math-1.5B-base and Qwen2.5-Math-7B-base (Yang et al., 2024) as the training base models. For the PRM, we mainly use Qwen2.5-Math-PRM-7B (Zhang et al., 2025) to generate process rewards. We also experiment with a weaker PRM, Skywork-PRM-1.5B (He et al., 2024b), to study the robustness of PROF to PRM quality. More details are provided in Appendix A. Model performance is evaluated on five benchmarks: Math500 (Hendrycks et al., 2021), Minerva Math (Lewkowycz et al., 2022), Olympiad Bench (He et al., 2024a), AMC2023111https://huggingface.co/datasets/math-ai/amc23, and AIME2024222https://huggingface.co/datasets/math-ai/aime24. We mainly use average@1616 for evaluation, i.e., accuracy averaged over 1616 responses per prompt under temperature 1.01.0. The models are allowed to generate 40964096 tokens.

4.2 Main Results

Model Algorithm Math500 Minerva Math Olympiad Bench AIME24 AMC23 Average
Qwen2.5-Math- 1.5B-base Base 39.9 11.4 19.1 3.5 23.6 19.5
GRPO 70.3 29.1 33.0 9.0 44.5 37.2
Blend 67.6 27.8 31.1 7.7 42.5 35.3
\cellcolorlightbluePROF-POS \cellcolorlightblue72.6 \cellcolorlightblue31.3 \cellcolorlightblue36.1 \cellcolorlightblue10.6 \cellcolorlightblue50.3 \cellcolorlightblue40.2
\cellcolorlightbluePROF-BOTH \cellcolorlightblue73.2 \cellcolorlightblue30.0 \cellcolorlightblue36.1 \cellcolorlightblue9.6 \cellcolorlightblue49.1 \cellcolorlightblue39.6
Qwen2.5-Math- 7B-base Base 42.0 12.8 19.2 12.9 30.0 23.4
GRPO 81.6 37.2 45.5 20.6 64.4 49.9
Blend 81.7 36.7 45.0 15.2 58.0 47.3
\cellcolorlightbluePROF-POS \cellcolorlightblue81.4 \cellcolorlightblue36.6 \cellcolorlightblue45.0 \cellcolorlightblue24.8 \cellcolorlightblue64.2 \cellcolorlightblue50.6
\cellcolorlightbluePROF-BOTH \cellcolorlightblue83.1 \cellcolorlightblue39.0 \cellcolorlightblue47.8 \cellcolorlightblue17.5 \cellcolorlightblue70.9 \cellcolorlightblue51.7
LLaMA-3.2- 3B-instruct t Base 30.0 8.8 6.1 2.3 10.6 11.6
GRPO 50.5 18.8 17.9 5.0 25.6 23.6
Blend 37.2 13.1 9.9 1.0 17.2 15.7
\cellcolorlightbluePROF-POS \cellcolorlightblue52.4 \cellcolorlightblue19.5 \cellcolorlightblue19.8 \cellcolorlightblue6.7 \cellcolorlightblue28.6 \cellcolorlightblue25.4
\cellcolorlightbluePROF-BOTH \cellcolorlightblue49.0 \cellcolorlightblue18.0 \cellcolorlightblue17.3 \cellcolorlightblue5.4 \cellcolorlightblue23.9 \cellcolorlightblue22.7
Table 2: Performance across Math500, Minerva Math, Olympiad Bench, AMC2023, and AIME2024. Blend denotes Blend-PRM-GRPO. Reported accuracy is average@16 under temperature 1.01.0, with each method tuned to its best setting.

We summarize our main results in Table 2, where Blend denotes a common way that mixes the PRM with outcome rewards (Zha et al., 2025; Cui et al., 2025; Zou et al., 2025). Following (Zou et al., 2025), the PRMs are averaged over steps for each response, weighted by a parameter β\beta, and added to outcome rewards. We use parameter β=0.8\beta=0.8 according to Table 5 of (Zou et al., 2025). Our main findings are as follows.

PROF Improves Accuracy under Standard and Matched-Cost Comparisons.

As shown in Table 2, our proposed methods, PROF-POS and PROF-BOTH, consistently outperform GRPO and Blend-PRM-GRPO across benchmarks and base models. For models starting from Qwen2.5-Math-1.5B-base, PROF-POS and PROF-BOTH achieve average accuracies of 40.2%40.2\% and 39.6%39.6\%, surpassing the standard GRPO baseline (37.2%37.2\%) and Blend-PRM-GRPO (35.3%35.3\%). A similar trend is observed with Qwen2.5-Math-7B-base, where PROF-POS and PROF-BOTH achieve 50.6%50.6\% and 51.7%51.7\% average accuracies, significantly above GRPO’s 49.9%49.9\% and Blend-PRM-GRPO’s 47.3%47.3\%. Moreover, for LLaMA-3.2-3B-instruct, whose policy distribution differs from the Qwen family, Blend performs even worse than GRPO, while PROF-POS still outperforms the baseline by 1.8%1.8\%. The learning dynamics in Figure 4 corroborate these findings, illustrating that PROF steadily maintains a consistent performance advantage over both GRPO and Blend-PRM-GRPO throughout training, with faster convergence and higher final accuracy than GRPO.

To further address efficiency and fairness concerns, we increase the rollout group size nn and policy update group size mm, and compare GRPO-n​16​m​16n16m16 with PROF-n​16​m​8n16m8 on Qwen2.5-Math-7B-base under matched compute cost. As shown in Figure 3, PROF achieves larger gains than GRPO at the same cost level. We compute average cost as Inference +3×+3\times Train ++ PRM, where the factor 33 is a rough FLOPs proxy for training relative to a forward pass. We further aggregate all five benchmarks by base-model pass rate into four difficulty levels: Level 1 (p>0.5p>0.5), Level 2 (0.25<p≤0.50.25<p\leq 0.5), Level 3 (0<p≤0.250<p\leq 0.25), and Level 4 (p=0p=0). PROF’s gain is especially pronounced on harder problems, plausibly because easier problems usually involve shorter and simpler reasoning with fewer flaws, making improvements smaller and more sensitive to PRM noise, whereas harder problems rely much more on PRM’s ability to distinguish trajectory quality. Due to space constraints, Figure 3 visualizes only Level 4 in the main text, while matched-cost curves for Levels 1–3 are provided in Appendix Figure 8.

Refer to caption
Refer to caption
Figure 3: Matched-cost comparison on Qwen2.5-Math-7B-base: overall average accuracy averaged over 55 benchmarks (left) and Level-4 hardest bucket (right). Average cost is computed as Inference +3×+3\times Train ++ PRM.
Refer to caption
Refer to caption
Figure 4: Learning dynamics from Qwen2.5-Math-1.5B-base (left) and Qwen2.5-Math-7B-base (right), compared with GRPO and Blend-PRM-GRPO. The y-axis is average@16 accuracy further averaged over Math500, Minerva Math, and Olympiad Bench.

Filtration Method is Much More Robust than Blending.

We plot the entropy loss and response length curves of GRPO, Blend-PRM-GRPO, and PROF in Figure 7. Blend-PRM-GRPO suffers from severe reward hacking because its entropy collapses quickly toward zero. Simultaneously, its response length in the right plot increases uncontrollably, indicating that the model has learned to game the PRM by over-generating verbose and repetitive steps to obtain a higher averaged process reward. As a result, Blend-PRM-GRPO’s test accuracy even falls below GRPO. In contrast, PROF maintains a gradual and slightly faster decrease in entropy loss together with controlled response-length growth. This illustrates that our filtration method effectively leverages the PRM signal while staying robust to reward hacking. Below, we further analyze the quality of intermediate reasoning steps.

4.3 PROF Improves Reasoning Process Quality

PROF Improves Reasoning Consistency.

To evaluate the quality of intermediate steps, we adopt Monte Carlo (MC) estimation, a common way to estimate the probability of reaching correct final answers (Wang et al., 2023; Xiong et al., 2024a; Luo et al., 2024). For this analysis, we select problem-response pairs from test prompts where our method and GRPO both produced the correct final answer. Both models were initialized from Qwen2.5-Math-7B-base. To estimate the value of each reasoning step, we generate eight independent completions from that point using a temperature of 1.0, and the resulting empirical success rate serves as the MC value. In Figure 5 (left), the average MC estimates across all five benchmarks are consistently higher for our model. The specific improvement gaps are 9.2%9.2\% on Math500, 37.4%37.4\% on Minerva Math, 15.9%15.9\% on Olympiad Bench, 9.2%9.2\% on AMC2023, and 11.1%11.1\% on AIME2024, which are much larger than the outcome-accuracy gap in Table 2.

PROF Reduces Flawed Reasoning within Correct Responses.

As a more direct faithfulness metric, we audit correct responses on the test set with Claude Sonnet 4.6 and ask whether the reasoning process contains any flaw (e.g., logical or arithmetic errors), even when the final answer is correct. The audit prompt is provided in Appendix A.2. In Figure 5 (right), the flawed-reasoning rate within correct responses decreases from 8%8\% for GRPO to 6%6\% for PROF. This complements Figure 1: Figure 1 measures flawed-reasoning prevalence in base-model outputs before RL (about 30%30\%, specifically 26.28%26.28\%), whereas Figure 5 reports the same notion after training, where both methods fall below 10%10\% and PROF remains lower. We also note that Claude-based auditing is still an approximate signal of reasoning quality and cannot fully replace careful human judgment on step granularity, subtle unsupported jumps, or the level of detail. Therefore, we additionally provide qualitative response comparisons in Figures 12 and 13. These examples consistently show that PROF produces concrete and verifiable intermediate deductions, GRPO tends to skip key steps, and Blend-PRM-GRPO is often verbose but less reliable in core calculations.

Refer to caption
Refer to caption
Figure 5: Reasoning-process quality of PROF vs. GRPO. Left: Monte Carlo step-value scores across five benchmarks. Right: flawed-reasoning rate within correct test responses judged by Claude Sonnet 4.6 (GRPO: 8%8\%, PROF: 6%6\%).

Additional process metrics on Math500 (step counts and averaged PRM scores) are moved to Appendix Figure 10. The key takeaway is that PROF improves process quality under both MC-based estimation and direct flaw auditing.

5 Ablations

5.1 Robustness to PRM Capability

Algorithm Math500 Minerva Math Olympiad Bench AIME24 AMC23 Average
GRPO 81.6 37.2 45.5 20.6 64.4 49.9
Blend (PRM-7B) 81.7 36.7 45.0 15.2 58.0 47.3
PROF (PRM-7B) 83.1 39.0 47.8 17.5 70.9 51.7
Blend (PRM-1.5B) 81.1 37.8 44.1 11.7 62.8 47.5
PROF-POS (PRM-1.5B) 82.9 39.4 47.4 19.2 66.1 51.0
PROF-BOTH (PRM-1.5B) 83.2 38.8 47.8 17.5 65.0 50.5
Table 3: Test accuracy (average@16, temperature 1.01.0) for Qwen2.5-Math-7B-base trained with different PRMs, averaged across five benchmarks.

To showcase PROF’s robustness to PRM quality, we use a weaker and smaller Skywork-PRM-1.5B (He et al., 2024b) while training from Qwen2.5-Math-7B-base. The results in Table 3 validate that when using a weaker PRM, Blend achieves lower accuracies, while PROF still maintains performance close to the model trained with the 7B PRM. This finding further corroborates the robustness of our algorithm.

5.2 Generality beyond GRPO: RAFT++

To demonstrate that PROF is a general filtration framework, we extend our experiments to RAFT++ (Xiong et al., 2025a), a rejection-sampling-based online training paradigm that only trains on positive samples. We compare PROF-Raft++ against standard RAFT++ baselines with different rollout budgets in Table 4. PROF-Raft++ not only outperforms the standard Raft++-n​4n4 baseline, but also significantly surpasses Raft++-n​8n8. Since RAFT++ only uses positive samples and does not involve negative samples, this comparison is primarily influenced by the number and quality of positive trajectories. Therefore, PROF’s priority-based filtration is algorithm-agnostic and consistently identifies high-quality reasoning paths that lead to better policy improvement, regardless of the underlying RL objective.

Method Average score
Raft++-n​4n4 35.27
Raft++-n​8n8 37.75
PROF-Raft++ (n​8​m​4n8m4) 39.29
Table 4: RAFT++ ablation under different rollout budgets.

5.3 Separating Correct and Incorrect

Refer to caption
Refer to caption
Refer to caption
Figure 6: Left two: averaged accuracy on Math500, Minerva Math, and Olympiad Bench for PROF variants from Qwen2.5-Math-1.5B-base and Qwen2.5-Math-7B-base. Right: reward gap after vs. before filtering, with and without separation.

We first test a no-separation variant (PROF w/o separation) that ranks all rollouts together. To mitigate PRM scale bias, we center each step score by subtracting the batch mean. Even with centering, the rightmost plot in Figure 6 shows that PROF w/o separation has over 2%2\% gap between rewards before and after filtering, indicating disproportionate removal of negative samples. A likely reason is that incorrect responses often contain several locally correct steps, which can inflate averaged PRM scores and blur process-outcome consistency. Separating correct and incorrect groups alleviates this bias.

We then compare three variants: PROF-POS (consistency filtering on correct group only), PROF-NEG (incorrect group only), and Filter-Random (random filtering on both groups) (Xu et al., 2025). As shown in Figure 6, PROF-POS and PROF-BOTH are the best-performing strategies across both 1.5B and 7B settings; PROF-BOTH is typically more sample-efficient, PROF-NEG is weaker, Filter-Random is only slightly above GRPO, and w/o separation is the worst. These results suggest that preserving quality in correct responses is the dominant factor, while consistency control on incorrect responses is secondary. More filtration ablations are provided in Appendix B.

This ablation highlights a practical trade-off between PROF-BOTH and PROF-POS. PROF-BOTH usually converges faster by using consistency signals from both groups, while PROF-POS can be more robust when PRM reliability is weaker or distribution shift is larger, since it avoids tightly shaping the incorrect group with noisy estimates. In both cases, improving correct trajectories is the main driver, and filtering incorrect trajectories mainly affects efficiency and stability.

6 Conclusion and Future Work

This work introduces Process Consistency Filter (PROF), a data curation technique that filters generated responses based on PRM–ORM consistency while maintaining a balanced correct/incorrect ratio. We demonstrate that PROF consistently improves final-answer accuracy and shapes the policy to generate more detailed and fine-grained intermediate reasoning steps. PROF is also a general filtration framework rather than one tied to a specific PRM or RL objective. Thus, using pre-trained PRMs in our experiments is not a limitation; instead, it highlights the robustness of our algorithm to different PRMs and suggests that training a task-specific PRM for each base model is unnecessary. Exploring stronger or more diverse PRMs, and extending PROF to other reasoning tasks such as coding (Jimenez et al., 2023) and web navigation (Zhou et al., 2023), remains important future work.

Broader Impact and Ethics Statement

Our work contributes to AI safety by enhancing the faithfulness and interpretability of chain-of-thought reasoning, mitigating the risk of misleading hallucinations. However, we acknowledge two potential risks. First, the reliance on oversampling and dense process reward computation increases computational overhead and environmental impact compared to standard baselines. Second, our filtration mechanism depends on pre-trained Process Reward Models (PRMs); if these PRMs harbor biases toward specific reasoning patterns or languages, our method may inadvertently amplify such biases by filtering out diverse but valid solutions. We encourage future research to address these efficiency and fairness challenges.

Limitations

Although PROF can effectively improve robustness to PRM noise and increase reasoning-step quality, our method requires more computation than Blend or vanilla GRPO because it first oversamples and then filters. How to balance efficiency and reasoning quality remains an important direction for future work. Finally, we acknowledge the use of AI assistants (e.g., ChatGPT) for grammatical error correction and polishing of the manuscript.

References

  • B. Baker, J. Huizinga, L. Gao, Z. Dou, M. Y. Guan, A. Madry, W. Zaremba, J. Pachocki, and D. Farhi (2025) Monitoring reasoning models for misbehavior and the risks of promoting obfuscation. arXiv preprint arXiv:2503.11926. Cited by: §1, §2.
  • E. Beeching, S. C. Huang, A. Jiang, J. Li, B. Lipkin, Z. Qina, K. Rasul, Z. Shen, R. Soletskyi, and L. Tunstall (2024) NuminaMath 7b cot. Numina Hugging Face. Note: https://huggingface.co/AI-MO/NuminaMath-7B-CoT Cited by: §4.1.
  • R. A. Bradley and M. E. Terry (1952) Rank analysis of incomplete block designs: i. the method of paired comparisons. Biometrika 39 (3/4), pp. 324–345. Cited by: §2.
  • H. Chen, K. Zheng, Q. Zhang, G. Cui, Y. Cui, H. Ye, T. Lin, M. Liu, J. Zhu, and H. Wang (2025a) Bridging supervised learning and reinforcement learning in math reasoning. arXiv preprint arXiv:2505.18116. Cited by: §2.
  • Y. Chen, J. Benton, A. Radhakrishnan, J. Uesato, C. Denison, J. Schulman, A. Somani, P. Hase, M. Wagner, F. Roger, et al. (2025b) Reasoning models don’t always say what they think. arXiv preprint arXiv:2505.05410. Cited by: §1, §2.
  • K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, et al. (2021) Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168. Cited by: §1.
  • G. Cui, L. Yuan, Z. Wang, H. Wang, W. Li, B. He, Y. Fan, T. Yu, Q. Xu, W. Chen, et al. (2025) Process reinforcement through implicit rewards. arXiv preprint arXiv:2502.01456. Cited by: §1, §4.2.
  • H. Dong, W. Xiong, D. Goyal, Y. Zhang, W. Chow, R. Pan, S. Diao, J. Zhang, K. Shum, and T. Zhang (2023) Raft: reward ranked finetuning for generative foundation model alignment. arXiv preprint arXiv:2304.06767. Cited by: §2.
  • H. Dong, W. Xiong, B. Pang, H. Wang, H. Zhao, Y. Zhou, N. Jiang, D. Sahoo, C. Xiong, and T. Zhang (2024) Rlhf workflow: from reward modeling to online rlhf. arXiv preprint arXiv:2405.07863. Cited by: §2.
  • A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Yang, A. Fan, et al. (2024) The llama 3 herd of models. arXiv e-prints, pp. arXiv–2407. Cited by: §1.
  • J. Eisenstein, C. Nagpal, A. Agarwal, A. Beirami, A. D’Amour, D. Dvijotham, A. Fisch, K. Heller, S. Pfohl, D. Ramachandran, et al. (2023) Helping or herding? reward model ensembles mitigate but do not eliminate reward hacking. arXiv preprint arXiv:2312.09244. Cited by: §2.
  • C. He, R. Luo, Y. Bai, S. Hu, Z. L. Thai, J. Shen, J. Hu, X. Han, Y. Huang, Y. Zhang, et al. (2024a) Olympiadbench: a challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems. arXiv preprint arXiv:2402.14008. Cited by: §4.1.
  • J. He, T. Wei, R. Yan, J. Liu, C. Wang, Y. Gan, S. Tu, C. Y. Liu, L. Zeng, X. Wang, B. Wang, Y. Li, F. Zhang, J. Xu, B. An, Y. Liu, and Y. Zhou (2024b) Skywork-o1 open series. Zenodo. External Links: Document, Link Cited by: §4.1, §5.1.
  • D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt (2021) Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874. Cited by: §4.1.
  • A. Jacovi, Y. Bitton, B. Bohnet, J. Herzig, O. Honovich, M. Tseng, M. Collins, R. Aharoni, and M. Geva (2024) A chain-of-thought is as strong as its weakest link: a benchmark for verifiers of reasoning chains. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Cited by: §2.
  • A. Jaech, A. Kalai, A. Lerer, A. Richardson, A. El-Kishky, A. Low, A. Helyar, A. Madry, A. Beutel, A. Carney, et al. (2024) Openai o1 system card. arXiv preprint arXiv:2412.16720. Cited by: §1.
  • C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. Narasimhan (2023) Swe-bench: can language models resolve real-world github issues?. arXiv preprint arXiv:2310.06770. Cited by: §6.
  • M. Khalifa, R. Agarwal, L. Logeswaran, J. Kim, H. Peng, M. Lee, H. Lee, and L. Wang (2025) Process reward models that think. arXiv preprint arXiv:2504.16828. Cited by: §2.
  • S. S. Kim, Q. V. Liao, M. Vorvoreanu, S. Ballard, and J. W. Vaughan (2024) " I’m not sure, but…": examining the impact of large language models’ uncertainty expression on user reliance and trust. In Proceedings of the 2024 ACM conference on fairness, accountability, and transparency, pp. 822–835. Cited by: §2.
  • A. Lewkowycz, A. Andreassen, D. Dohan, E. Dyer, H. Michalewski, V. Ramasesh, A. Slone, C. Anil, I. Schlag, T. Gutman-Solo, et al. (2022) Solving quantitative reasoning problems with language models. Advances in neural information processing systems 35, pp. 3843–3857. Cited by: §4.1.
  • H. Lightman, V. Kosaraju, Y. Burda, H. Edwards, B. Baker, T. Lee, J. Leike, J. Schulman, I. Sutskever, and K. Cobbe (2023) Let’s verify step by step. In The Twelfth International Conference on Learning Representations, Cited by: §1, §2, §2.
  • Y. Lin, H. Lin, W. Xiong, S. Diao, J. Liu, J. Zhang, R. Pan, H. Wang, W. Hu, H. Zhang, et al. (2023) Mitigating the alignment tax of rlhf. arXiv preprint arXiv:2309.06256. Cited by: §2.
  • L. Luo, Y. Liu, R. Liu, S. Phatale, M. Guo, H. Lara, Y. Li, L. Shu, Y. Zhu, L. Meng, J. Sun, and A. Rastogi (2024) Improve mathematical reasoning in language models by automated process supervision. External Links: 2406.06592, Link Cited by: §4.3.
  • Q. Lyu, S. Havaldar, A. Stein, L. Zhang, D. Rao, E. Wong, M. Apidianaki, and C. Callison-Burch (2023) Faithful chain-of-thought reasoning. In The 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (IJCNLP-AACL 2023), Cited by: §1, §2.
  • E. J. Michaud, A. Gleave, and S. Russell (2020) Understanding learned reward functions. arXiv preprint arXiv:2012.05862. Cited by: §1.
  • M. Nguyen, L. Luo, F. Shiri, D. Phung, Y. Li, T. Vu, and G. Haffari (2024) Direct evaluation of chain-of-thought in multi-hop reasoning with knowledge graphs. In Findings of the Association for Computational Linguistics: ACL 2024, Cited by: §2.
  • D. Paul, R. West, A. Bosselut, and B. Faltings (2024) Making reasoning matter: measuring and improving faithfulness of chain-of-thought reasoning. In Findings of the Association for Computational Linguistics: EMNLP 2024, Cited by: §2.
  • J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov (2017) Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347. Cited by: §3.
  • Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. Li, Y. Wu, et al. (2024) Deepseekmath: pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300. Cited by: §1, §3.
  • G. Sheng, C. Zhang, Z. Ye, X. Wu, W. Zhang, R. Zhang, Y. Peng, H. Lin, and C. Wu (2025) Hybridflow: a flexible and efficient rlhf framework. In Proceedings of the Twentieth European Conference on Computer Systems, pp. 1279–1297. Cited by: §A.1.
  • J. Tien, J. Z. He, Z. Erickson, A. D. Dragan, and D. S. Brown (2022) Causal confusion and reward misidentification in preference-based reward learning. arXiv preprint arXiv:2204.06601. Cited by: §1.
  • M. Turpin, J. Michael, E. Perez, and S. R. Bowman (2023) Language models don’t always say what they think: unfaithful explanations in chain-of-thought prompting. In Advances in Neural Information Processing Systems 36 (NeurIPS 2023), Cited by: §2.
  • P. Wang, L. Li, Z. Shao, R. Xu, D. Dai, Y. Li, D. Chen, Y. Wu, and Z. Sui (2023) Math-shepherd: verify and reinforce llms step-by-step without human annotations. arXiv preprint arXiv:2312.08935. Cited by: §2, §4.3.
  • W. Xiong, C. Shi, J. Shen, A. Rosenberg, Z. Qin, D. Calandriello, M. Khalman, R. Joshi, B. Piot, M. Saleh, et al. (2024a) Building math agents with multi-turn iterative preference learning. arXiv preprint arXiv:2409.02392. Cited by: §2, §4.3.
  • W. Xiong, J. Yao, Y. Xu, B. Pang, L. Wang, D. Sahoo, J. Li, N. Jiang, T. Zhang, C. Xiong, et al. (2025a) A minimalist approach to llm reasoning: from rejection sampling to reinforce. arXiv preprint arXiv:2504.11343. Cited by: §2, §5.2.
  • W. Xiong, H. Zhang, N. Jiang, and T. Zhang (2024b) An implementation of generative prm. Cited by: §2.
  • W. Xiong, H. Zhang, C. Ye, L. Chen, N. Jiang, and T. Zhang (2025b) Self-rewarding correction for mathematical reasoning. arXiv preprint arXiv:2502.19613. Cited by: §1.
  • W. Xiong, W. Zhao, W. Yuan, O. Golovneva, T. Zhang, J. Weston, and S. Sukhbaatar (2025c) StepWiser: stepwise generative judges for wiser reasoning. External Links: 2508.19229, Link Cited by: §2.
  • Y. E. Xu, Y. Savani, F. Fang, and Z. Kolter (2025) Not all rollouts are useful: down-sampling rollouts in llm reinforcement learning. arXiv preprint arXiv:2504.13818. Cited by: §2, §3, §5.3.
  • A. Yang, B. Zhang, B. Hui, B. Gao, B. Yu, C. Li, D. Liu, J. Tu, J. Zhou, J. Lin, et al. (2024) Qwen2. 5-math technical report: toward mathematical expert model via self-improvement. arXiv preprint arXiv:2409.12122. Cited by: §1, §2, §2, §4.1.
  • W. J. Yeo, R. Satapathy, R. S. M. Goh, and E. Cambria (2024) How interpretable are reasoning explanations from prompting large language models?. In Findings of the Association for Computational Linguistics: NAACL 2024, Cited by: §1, §2.
  • P. Yu, W. Yuan, O. Golovneva, T. Wu, S. Sukhbaatar, J. Weston, and J. Xu (2025a) Rip: better models by survival of the fittest prompts. arXiv preprint arXiv:2501.18578. Cited by: §2.
  • Q. Yu, Z. Zhang, R. Zhu, Y. Yuan, X. Zuo, Y. Yue, W. Dai, T. Fan, G. Liu, L. Liu, et al. (2025b) Dapo: an open-source llm reinforcement learning system at scale. arXiv preprint arXiv:2503.14476. Cited by: §A.1, §2.
  • W. Yuan, R. Y. Pang, K. Cho, S. Sukhbaatar, J. Xu, and J. Weston (2024) Self-rewarding language models. arXiv preprint arXiv:2401.10020 3. Cited by: §2.
  • K. Zha, Z. Gao, M. Shen, Z. Hong, D. S. Boning, and D. Katabi (2025) RL tango: reinforcing generator and verifier together for language reasoning. arXiv preprint arXiv:2505.15034. Cited by: §1, §4.2.
  • C. Zhang, W. Shen, L. Zhao, X. Zhang, L. Qi, W. Dou, and J. Bian (2024) Policy filtration in rlhf to fine-tune llm for code generation. Cited by: §2.
  • Z. Zhang, C. Zheng, Y. Wu, B. Zhang, R. Lin, B. Yu, D. Liu, J. Zhou, and J. Lin (2025) The lessons of developing process reward models in mathematical reasoning. arXiv preprint arXiv:2501.07301. Cited by: §1, §2, §3, §4.1.
  • J. Zhao, R. Liu, K. Zhang, Z. Zhou, J. Gao, D. Li, J. Lyu, Z. Qian, B. Qi, X. Li, and B. Zhou (2025) GenPRM: scaling test-time compute of process reward models via generative reasoning. External Links: 2504.00891, Link Cited by: §2.
  • C. Zheng, Z. Zhang, B. Zhang, R. Lin, K. Lu, B. Yu, D. Liu, J. Zhou, and J. Lin (2024) Processbench: identifying process errors in mathematical reasoning. arXiv preprint arXiv:2412.06559. Cited by: §2, §3.
  • S. Zhou, F. F. Xu, H. Zhu, X. Zhou, R. Lo, A. Sridhar, X. Cheng, T. Ou, Y. Bisk, D. Fried, et al. (2023) Webarena: a realistic web environment for building autonomous agents. arXiv preprint arXiv:2307.13854. Cited by: §6.
  • D. Zhu, X. Wei, G. Zhao, W. Wu, H. Zou, J. Ran, X. Wang, L. Sun, X. Zhang, and S. Li (2025) Chain-of-thought matters: improving long-context language models with reasoning path supervision. arXiv preprint arXiv:2502.20790. Cited by: §1.
  • J. Zou, L. Yang, J. Gu, J. Qiu, K. Shen, J. He, and M. Wang (2025) ReasonFlux-prm: trajectory-aware prms for long chain-of-thought reasoning in llms. arXiv preprint arXiv:2506.18896. Cited by: §1, §3, §4.2.

Appendix A Additional Experimental Details and Results

A.1 Main Experiments

The implementations are based on the verl framework (Sheng et al., 2025), and we follow most of its parameter settings. Specifically, we use the AdamW optimizer with learning rate 1×10−61\times 10^{-6}. We adopt the clip-higher trick (Yu et al., 2025b), which clips the sampling ratio πθ/πold\pi_{\theta}/\pi_{\text{old}} to an asymmetric range (1−ϵlow,1+ϵhigh)(1-\epsilon_{\text{low}},1+\epsilon_{\text{high}}). Specifically, we set ϵlow=0.2,ϵhigh=0.28\epsilon_{\text{low}}=0.2,\epsilon_{\text{high}}=0.28 for models initialized from Qwen2.5-Math-1.5B-base and maintain ϵhigh=ϵlow=0.2\epsilon_{\text{high}}=\epsilon_{\text{low}}=0.2 for other cases. In each iteration, we sample 10241024 prompts and roll out n=4n=4 responses per prompt for GRPO and n=8n=8 responses for PROF. Note that the policy update number for all algorithms is m=4m=4. For the regularization of step numbers in Algorithm 1, we take λ=10\lambda=10 and Hλ=30H_{\lambda}=30. For the rollout stage, we use a temperature of 1.01.0 and a top-p value of 1.01.0. We set the KL loss coefficient to 0.0010.001 and entropy loss coefficient to 0.0010.001. All the models are trained with 88 H100 GPUs. We set the training mini-batch size as 256256 and allow the models to generate 40964096 tokens per prompt.

Refer to caption
Refer to caption
Figure 7: Entropy loss (left) and response length (right) of the models initialized from Qwen2.5-Math-7B-base.

A.2 Prompt Template

We present the template used by the LLM to audit whether a correct response still contains reasoning flaws in Table 5.

Prompt for Auditing Reasoning Flaws in a Candidate Solution System Your task is to audit a candidate solution to a math problem and determine whether the reasoning process contains any flaw or error, even if the final answer is correct. User You must evaluate the solution step by step and focus on the validity of the reasoning process, not writing style. A solution can be flawed even when the final answer is correct. Reasoning flaws include: • random guessing; • skipping or jumping over intermediate steps; • logical errors; • arithmetic errors; • misuse of definitions, theorems, or formulas. Do not penalize: minor wording issues or overly specific solution styles. Important rules: • Judge the reasoning process itself, not just the final answer. • If the final answer is correct but any earlier step is invalid or unsupported, mark the reasoning as flawed. • Identify the earliest step where the reasoning first becomes flawed. • Once an earliest flawed step is found, later steps may be unreliable because they can depend on that error. You will be given a math problem and a candidate solution. If the candidate solution does not explicitly label steps, first segment it into natural reasoning steps before judging. Return your judgment in exactly the following format: <final_answer_correct>yes / no / unclear</final_answer_correct>
<reasoning_flawed>yes / no</reasoning_flawed>
<earliest_flawed_step>step number or none</earliest_flawed_step>
<error_type>arithmetic | algebra | logical | unsupported_jump | random_guessing | theorem_misuse | missing_case | contradiction | answer_first_rationalization | none</error_type>
<brief_explanation>
A concise explanation of why the reasoning is valid, or why the earliest flawed step is flawed. </brief_explanation>
Now evaluate the following example. [Problem]
{{prompt}}
[Candidate Solution]
{{responses}}
Table 5: Prompt for auditing flawed reasoning in candidate solutions via LLM-as-a-judge.

Appendix B Additional Experimental Results

In this section, we include additional ablation studies and evaluation results for a more comprehensive understanding of the PROF framework.

B.1 Matched-Cost Results for Difficulty Levels 1–3

Refer to caption
Refer to caption
Refer to caption
Figure 8: Matched-cost comparison on Qwen2.5-Math-7B-base for difficulty Level 1 (p>0.5p>0.5), Level 2 (0.25<p≤0.50.25<p\leq 0.5), and Level 3 (0<p≤0.250<p\leq 0.25). Average cost is computed as Inference +3×+3\times Train ++ PRM.

B.2 Effect of Rollout Numbers

Refer to caption
Figure 9: The averaged accuracy across all five benchmarks over rollout sizes n=4,8,12,16n=4,8,12,16 for filtering both correct and incorrect groups with PRM consistency (Both) and only the correct group with PRM consistency (Correct).

We study the scale of rollout numbers nn with fixed policy-update number m=4m=4 by varying n=4,8,12,16n=4,8,12,16. The lower-right plot in Figure 9 presents the test accuracy averaged over all five benchmarks for PROF-BOTH (Both) and PROF-POS (Correct) initialized from Qwen2.5-Math-7B-base. We observe that performance first increases and then decreases as nn grows, revealing a trade-off between enhancing process reasoning quality and avoiding reward hacking. Notably, PROF-POS decreases later (after n=12n=12) because it only leverages PRM influence in the correct group, indicating that PROF-POS is more robust when PRM influence becomes stronger, such as when the ranking-and-filtering scale increases.

B.3 Additional Process Metrics on Math500

Refer to caption
Refer to caption
Figure 10: Supplementary process metrics on Math500 for PROF vs. GRPO: number of reasoning steps (left) and averaged Qwen2.5-Math-PRM-7B scores (right).

B.4 Variants of Filtration Methods

Algorithm Math500 Minerva Math Olympiad Bench AIME24 AMC23 Average
Mean 83.1 39.0 47.8 17.5 70.9 51.7
Minimum 82.9 38.3 46.7 20.8 65.9 50.9
Sum 82.4 38.1 47.4 17.7 67.5 50.6
Ratio 81.4 36.6 45.0 24.8 65.2 50.6
Table 6: Performance of different filtration ways in PROF starting from Qwen2.5-Math-7B-base.

In this subsection, we investigate different ways of computing the consistency score rpror^{\mathrm{pro}}, in addition to taking the mean of PRM scores over steps. Here, Mean denotes averaging over steps in Algorithm 1; Minimum and Sum denote taking the minimum and the sum over steps; and Ratio denotes filtering while preserving the original positive/negative sample distribution instead of balancing it. As shown in Table 6, Minimum (50.9%50.9\%), Sum (50.6%50.6\%), and Ratio (50.6%50.6\%) all underperform Mean. This suggests that the mean provides a more stable estimate of reasoning consistency: unlike the minimum, it is less sensitive to a single poorly scored step, and unlike the sum, it avoids bias toward longer trajectories. Additionally, balancing the correct/incorrect ratio lets consistency-based filtering select the better-supported group without breaking class balance.

B.5 Effect of Step Number

To verify that PROF does not help merely by increasing the number of steps, we evaluate Filter-Nstep, which ranks and filters samples by shorter step counts instead of lower PRM–ORM consistency.

From Table 6, we find that Ratio scores only 50.6%50.6\% on average and cannot compete with balanced filtering (PROF), which further corroborates the importance of maintaining a balanced correct/incorrect proportion. Additionally, because PROF increases the number of intermediate reasoning steps, we compare against simple step-length filtering to verify that the gain does not come merely from longer responses. As shown in Figure 11 and Table 7, Filter-Nstep mainly manipulates step length, exhibits an unreasonable increase followed by a sudden drop, and yields inferior average accuracy.

Algorithm Math500 Minerva Math Olympiad Bench AIME24 AMC23 Average
PROF-BOTH 83.1 39.0 47.8 17.5 70.9 51.7
Filter-Nstep 81.5 35.5 45.9 16.3 58.6 47.6
Table 7: Performance of filtration variants besides PROF-BOTH for Qwen2.5-Math-7B-base, averaged over all five benchmarks. Ratio preserves the original correct/incorrect proportion, and Filter-Nstep ranks and filters by the number of step segments.
Refer to caption
Figure 11: The number of reasoning steps during training time for PROF-GRPO and Filter-Nstep initialized from Qwen2.5-Math-7B-base.

Appendix C Additional Examples

Refer to caption
Figure 12: A Minerva-Math example comparing distinct intermediate reasoning patterns of PROF-BOTH, vanilla GRPO, and Blend-PRM-GRPO. PROF-BOTH presents concrete and correct deduction steps. GRPO’s solution skips detailed deductions and contains flaws in calculation precision and final rounding. Blend-PRM-GRPO is long-winded and makes a major mistake in computing the power.
Refer to caption
Figure 13: A Math500 example to compare distinct intermediate reasoning patterns of PROF-BOTH, vanilla GRPO and Blend-PRM-GRPO. PROF-BOTH presents concrete and correct deduction steps. PROF-BOTH’s solution shows how to find the divisors and summation in detail, and is easy to follow. GRPO skips all core reasoning. Blend-PRM-GRPO has inefficient and excessively tedious steps.