## Source: https://arxiv.org/pdf/2510.23038 # Incentivizing Agentic Reasoning in LLM Judges via Tool-Integrated Reinforcement Learning 2025-10-28 # Incentivizing Agentic Reasoning in LLM Judges via Tool-Integrated Reinforcement Learning Ran Xu 1,2* , Jingjing Chen 2, Jiayu Ye 2, Yu Wu 2, Jun Yan 3, Carl Yang 1 and Hongkun Yu 2 > 1 Emory University, 2Google, 3Google Cloud AI Research, *Work done during an internship at Google. Large Language Models (LLMs) are widely used as judges to evaluate response quality, providing a scalable alternative to human evaluation. However, most LLM judges operate solely on intrinsic text-based reasoning, limiting their ability to verify complex constraints or perform accurate computation. Motivated by the success of tool-integrated reasoning (TIR) in numerous tasks, we propose TIR-Judge ,an end-to-end RL framework for training LLM judges that integrates a code executor for precise evaluation. TIR-Judge is built on three principles: (i) diverse training across verifiable and non-verifiable domains, (ii) flexible judgment formats (pointwise, pairwise, listwise), and (iii) iterative RL that bootstraps directly from the initial model without distillation. On seven public benchmarks, TIR-Judge surpasses strong reasoning-based judges by up to 6.4% (pointwise) and 7.7% (pairwise), and achieves listwise performance comparable to Claude-Opus-4 despite having only 8B parameters. Remarkably, TIR-Judge -Zero – trained entirely without distilled judge trajectories, matches the performance of distilled variants, demonstrating that tool-augmented judges can self-evolve through iterative reinforcement learning. Keywords: LLM-as-a-judge, Reinforcement Learning, Tool-Integrated Reasoning 1. Introduction Large Language Model (LLM)-based judges are emerging as a critical component in the LLM ecosystem, typically used with scoring and ranking model outputs. This evaluation capability is essential at multiple stages of LLM development: during post-training, judges provide preference signals for alignment (Chen et al., 2025a; Whitehouse et al., 2025); at inference time, judges verify and select responses through best-of-N decoding (Huang et al., 2025a); and during evaluation, judges deliver reliable assessments without manual human assessment (Li et al., 2024a). Thus, training accurate LLM-based judges is of great importance for building powerful language models. Write a poem that's at least 350 words about the beauty of eucalyptus trees and their many uses. > Where desert heat whispers and mountains rise tall, The eucalyptus stands, a majestic call, …… > The response contains 392 words, which meets the requirements. > ```python > response ="""Where desert heat whispers ……""" print(len(response.split()) > ``` ```output > 321 > ``` The response only contains 321 words, which does not meet the requirements. Text-based LLM judge > Response > User Prompt > Tool-integrated LLM judge Figure 1 | An example of LLM judge augmented with code execution, enabling precise judgments. Classical evaluation with reward models often outputs scores directly, which cannot fully harvest the inherent reasoning capa-bility of LLM. Recent progress in generative reward modeling (Zhang et al., 2025; Zhao et al., 2025) and reinforcement learning equips judges with thinking before produc-ing final predictions (Chen et al., 2025b; Guo et al., 2025b; Hong et al., 2025; White-house et al., 2025). While these approaches enhance judge quality by equipping LLMs with long chains of textual reasoning traces, they remain inherently limited in scenarios that require precise computation or symbolic reasoning – capabilities that are much more challenging for text-only models (Mirzadeh et al., 2025). Recent advances in LLM tool-use provide a promising avenue to overcome the limitations of Corresponding author(s): ranxu@google.com © 2025 Google. All rights reserved > arXiv:2510.23038v1 [cs.CL] 27 Oct 2025 Incentivizing Agentic Reasoning in LLM Judges via Tool-Integrated Reinforcement Learning text-only judges (Chen et al., 2023; Gao et al., 2023). By granting access to executable interfaces for enumeration, verification, and computation, tools enable exact validation of reasoning steps rather than relying on potentially error-prone text-based inference. For example, code execution can automatically verify outputs on certain instructions (Zhou et al., 2023) (as shown in Figure 1) or check intermediate calculations in math reasoning (Xiong et al., 2025). Early attempts have also explored equipping LLM judges with tool-use abilities (Findeis et al., 2025; Li et al., 2024b; Peng et al., 2025), but these approaches reveal two major limitations. (i) Inference-time restriction : most methods integrate tool-use only at the inference stage, preventing deeper integration between reasoning processes and tool execution. (ii) Narrow task coverage : many are tailored to specific domains or specialized task types, which limits their applicability in general-purpose judging scenarios. These gaps highlight the need for robust judges that tightly couple reasoning with tool execution and be optimized end-to-end. Motivated by these challenges, our goal is to develop an LLM judge that can reliably integrate reasoning with code interpreter execution. Incorporating tool-integrated reasoning (TIR) (Feng et al., 2025; Li et al., 2025b; Lin and Xu, 2025; Xue et al., 2025), we propose TIR-Judge , a framework that leverages reinforcement learning (RL) to teach models to generate code, execute it with interpreters, and iteratively refine their reasoning based on the resulting outputs. By reinforcing this cycle of reasoning and tool-use, TIR-Judge equips LLM judges with the ability at the training time to deliver more accurate and verifiable evaluations across diverse tasks. Then, to fully unleash the potential of RL for TIR-Judge , we introduce several key design choices. (i) Task diversity : To balance between different tasks, we construct training prompts spanning both verifiable domains (e.g., competitive programming, mathematical reasoning) and non-verifiable domains (e.g., dialogue, safety, general coding), allowing the model to learn when tool invocation is beneficial and when pure reasoning suffices. (ii) Judgment flexibility : To accommodate to different input/output formats, we diversify the evaluation tasks to cover pointwise, pairwise, and listwise ranking, ensuring broad applicability across practical use cases. (iii) Data efficiency : unlike prior methods that rely on distillation as cold-start for RL (Chen et al., 2025b; Hong et al., 2025), we demonstrate that TIR-Judge can bootstrap from the initial checkpoint. Specifically, TIR-Judge -Zero trains purely with iterative reinforcement learning for achieving self-improvement, while TIR-Judge -Distill provides an optional variant using a small amount of distillation data. Our contribution can be summarized as follows: • We introduce TIR-Judge , a tool-integrated framework for training LLM-based judges with end-to-end multi-turn reinforcement learning. To the best of our knowledge, this is the first approach that jointly optimizes reasoning and tool-use for training LLM-based judges via RL. • We design several key strategies to fully exploit the power of reinforcement learning, including task diversification across verifiable and non-verifiable domains, flexible judgment formats (pointwise, pairwise, listwise), as well as an iterative RL scheme that unlocks self-improvement in tool use even without distillation. • We evaluate TIR-Judge on seven public benchmarks covering diverse tasks and input formats. TIR-Judge consistently outperforms strong reasoning-based judges, achieving gains of up to 6.4% (pointwise) and 7.7% (pairwise). Moreover, TIR-Judge shows strong parameter efficiency: With only 8B parameters, it surpasses the 32B reasoning reward models on the PPE dataset, and reaches 96% of the performance of Claude-Opus-4 in the listwise setting in RewardBench 2. Interestingly, TIR-Judge -Zero, the judge trained without any distillation, achieves a 1.2% gain over its distilled counterpart at 4B scale, highlighting the power of RL to bootstrap reasoning and tool-use capabilities. > 2Incentivizing Agentic Reasoning in LLM Judges via Tool-Integrated Reinforcement Learning 2. Related Works Reasoning-Enhanced Reward and Judge Models. A growing line of work strengthens reward models (RMs) and judges by explicitly training them to reason before issuing a score or decision. Generative Verifiers (Zhang et al., 2025) frame verification as next-token prediction with chain-of-thought, achieving notable gains on mathematical and algorithmic reasoning tasks. Other approaches strengthen judgment quality by generating critiques prior to reward prediction (Ankner et al., 2024; Shen et al., 2024; Yu et al., 2025b), leveraging multi-round preference optimization to progressively refine judge capability (Wang et al., 2024b), or teaching models to first propose evaluation plans or rubrics before producing a final judgment (Liu et al., 2025a; Saha et al., 2025). Liu et al. (2025c) further study how to allocate additional compute and structure critique signals to improve reliability. More recently, reinforcement learning has been used to optimize LLM-as-a-Judge (Chen et al., 2025a,b; Guo et al., 2025b; Hong et al., 2025; Whitehouse et al., 2025), encouraging longer, higher-quality reasoning and reducing bias across pairwise and pointwise settings (Whitehouse et al., 2025). While effective at strengthening textual reasoning and planning, these methods remain limited to reasoning expressed in natural language and often focus primarily on pairwise judgment. Tool-Assisted Reward and Judge Models. Another line of work augments judges with external tools. Li et al. (2024b) incorporate verifiable signals alongside preference data for judge training, though primarily within tool-use scenarios. Zhuge et al. (2025) evaluate agentic judge capabilities in agent settings, and Agentic Reward Modeling (Peng et al., 2025) integrates human preferences with correctness checks to construct more reliable rewards. Findeis et al. (2025) study whether external tools (e.g., code execution, web search) improve LLM-as-a-Judge annotations, reporting consistent but task-dependent gains. However, these approaches largely depend on prompted tool use rather than training judges to learn when and how to invoke tools and to integrate their outputs into decisions. Reinforcement Learning for Tool-integrated Reasoning. Recent work explores reinforcement learning for TIR. Bai et al. (2025); Feng et al. (2025); Li et al. (2025b) train LLMs to interleave reasoning with code execution, discovering strategic tool-use policies that improve math and program-ming tasks. Several works extend this paradigm by interleaving reasoning steps on search agents (Jin et al., 2025; Song et al., 2025; Xu et al., 2025), web agents (Qi et al., 2025; Zhuang et al., 2025) and coding agents (Du et al., 2025). Other studies focus on optimal reward design for TIR (Dong et al., 2025; Wang et al., 2025a) or provide theoretical analysis of its advantages (Lin and Xu, 2025). 3. Preliminaries Problem Setup. We consider the task of LLM-based judgment : given a user prompt 𝑥 ∈ X and 𝑛 model-generated responses Y = { 𝑦 1, 𝑦 2, . . . , 𝑦 𝑛 }, the goal is to evaluate the quality of responses for the prompt. The judge model 𝐽 𝜃 produces an evaluation output conditioned on (𝑥, Y) . In this work, we consider three evaluation settings: (i) Pointwise evaluation : given (𝑥, 𝑦 ), the judge assigns a scalar score, 𝐽 𝜃 (𝑥, 𝑦 ) = 𝑠 𝜃 (𝑥, 𝑦 ) ∈ ℝ; (ii) Pairwise evaluation : given (𝑥, 𝑦 𝑎 , 𝑦 𝑏 ), the judge selects the preferred response, 𝐽 𝜃 (𝑥, 𝑦 𝑎 , 𝑦 𝑏 ) = arg max 𝑖 ∈ { 𝑎,𝑏 } 𝑠 𝜃 (𝑥, 𝑦 𝑖 ), where 𝑠 𝜃 denotes a learned scoring function. This is also the most common evaluation setting; (iii) Listwise evaluation : given (𝑥, Y) with 𝑛 > 2,the judge returns the index of the best response, 𝐽 𝜃 (𝑥, Y) = arg max 𝑖 ∈ { 1,...,𝑛 } 𝑠 𝜃 (𝑥, 𝑦 𝑖 ). These settings unify a broad range of evaluations under a common framework 1. Tool-Augmented Judge. We extend the judge with the ability to call an external Python execution environment I. For the prompt 𝑥 ∈ X , At step 𝑘 , the judgment trajectory 𝑠 𝑘 is represented as 𝑠 𝑘 = {𝑟 1, 𝑐 1, 𝑜 1, . . . , 𝑟 𝑘 , 𝑐 𝑘 , 𝑜 𝑘 }, where 𝑟 𝑖 is a natural language reasoning step, 𝑐 𝑖 is a generated code, > 1Note that in our work, the reference answer is unseen during evaluation, different from the verification setting (Li et al., 2025a; Yan et al., 2025) where the reference answer is also a part of the input. > 3 Incentivizing Agentic Reasoning in LLM Judges via Tool-Integrated Reinforcement Learning Poem form: ✅, it is written in verse. > Topic: ✅, it highlights the eucalyptus beauty and eucalyptus uses. > Length requirement: Need python code to evaluate. > ```python > response = """Where desert heat whispers ……""" print(len(response.split()) > ``` > ```output > 321 > ``` > From the result, the response only contains 321 words. ❌Fails. > Final Verdict: The response does not follow the user prompt completely. > RL > Rejection Sampling > ●Final answer ✅ > ●Output format ✅ > ●Code execution ✅ > SFT RL > SFT Data SFT Data SFT Data > SFT Data SFT Data SFT Data > SFT RL Iteratively update > Training Data Training Data Training Data > SFT RL SFT Data SFT Data SFT Data > Rejection Sampling > TIR-Judge-Distill > TIR-Judge-Zero > 10 6 > chosen rejected > B > rejected rejected > chosen rejected > … > Pairwise Listwise > Pointwise > Execution in sandbox Tool-Integrated Reasoning Judge Evaluation Training Strategies for RL Figure 2 | Overall framework of TIR-Judge variants. TIR-Judge natively supports tool use during judgment and is designed to handle diverse input formats. and 𝑜 𝑖 = 𝐼 (𝑐 𝑖 ) is the execution result of 𝑐 𝑖 (Li et al., 2025b). The iterative process is defined as: (𝑟 𝑘 , 𝑐 𝑘 ) ∼ 𝐽 (𝑥 ⊕ 𝑠 𝑘 −1), 𝑜 𝑘 = I ( 𝑐 𝑘 ), 𝑠 𝑘 = 𝑠 𝑘 −1 ⊕ 𝑟 𝑘 ⊕ 𝑐 𝑘 ⊕ 𝑜 𝑘 . (1) This cycle continues until the judge produces a final prediction 𝑎 𝑖 ∼ 𝐽 (𝑥 ⊕ 𝑠 𝑇 ) with 𝑇 being the final step. Unlike traditional text-only reasoning, the trajectory now interleaves reasoning, code execution, and tool feedback, enabling the judge to ground its decision in verifiable evidence. 4. Training TIR-Judge We now describe the training procedure for TIR-Judge , which consists of four components: (1) data collection and filtering for RL, (2) the RL framework for training judges with integrated code execution tools, (3) reward design for RL, and (4) cold-start and iterative training strategies in RL. The overall framework of TIR-Judge is exhibited in Figure 2. 4.1. Data Collection and Filtering High-quality training data is crucial for RL with tool-augmented judges. Since judgment requires both prompts and candidate responses, we curate a collection of (prompt , responses ) tuples spanning multiple tasks. Our corpus integrates both human-annotated preference data and automatically generated synthetic pairs to ensure diversity and scalability. Real Preference Pairs. We sample human-labeled preference pairs from a variety of domains: general helpfulness — HelpSteer 3 (Wang et al., 2025b); reasoning — UltraInteract (Yuan et al., 2025), S1 (Muennighoff et al., 2025); coding — CodeRM (Ma et al., 2025b); instruction following (IF) — preference pairs from Tulu 3 (Lambert et al., 2024); safety — Safe-RLHF (Dai et al., 2024). Each prompt is paired with one preferred ( chosen ) response and one or more rejected responses. Synthetic Preference Pairs. Because reasoning preference data is often limited in scale, we augment the corpus with synthetic preference pairs generated from verifiable prompts. For each prompt, we sample responses from multiple open-source models, including Qwen3-8B/14B (Team, 2025), Gemma-2-9B (Team et al., 2024), and Gemma-3-12B (Team et al., 2025). The responses are automatically evaluated against verifiable functions (for IF tasks) or ground-truth solutions (for reasoning tasks) to form preference pairs. For IF , we use verifiable prompts from Tulu-3 (Lambert et al., 2024), where correctness can be programmatically verified using lexical or structural constraints. 4Incentivizing Agentic Reasoning in LLM Judges via Tool-Integrated Reinforcement Learning For reasoning , we employ MATH (Hendrycks et al., 2021) and DAPO-Math (Yu et al., 2025a) for math domain and WebInstruct (Ma et al., 2025a), and Loong (Huang et al., 2025b) for general domain, both of which provide ground-truth solutions for exact verification. In total, our dataset comprises approximately 26k preference pairs, including pointwise, pairwise, and listwise annotations, covering diverse domains such as helpfulness, reasoning, coding, safety, and verifiable instruction following. We apply strict 8-gram decontamination to eliminate any overlap between training prompts and evaluation benchmarks (Oren et al., 2024). This diverse mixture of data provides a strong foundation for training robust tool-augmented judges. 4.2. Tool-Integrated RL with Verifiable Rewards Overall Framework. We adopt DAPO (Yu et al., 2025a), an improved variant of GRPO (Shao et al., 2024), for training the LLM judge 𝐽 parameterized by 𝜋 𝜃 . Given a prompt–answer pair (𝑞, 𝑎 ), we first sample a group of 𝐺 rollouts {𝑠 𝑖 }𝐺 𝑖 =1 from the current policy 𝜋 𝜃 old . Each rollout 𝑠 𝑖 is assigned a scalar reward 𝑅 𝑖 = 𝑅 (𝑠 𝑖 , 𝑎 ) with access to the oracle answer 𝑎 . The policy 𝜋 𝜃 is then updated with the following clipped policy gradient objective: J ( 𝜃 ) = 𝔼 (𝑞,𝑎 )∼D ,{𝑠 𝑖 }𝐺 𝑖 =1∼𝜋 𝜃 old (· | 𝑞 ) 1 Í𝐺 𝑖 =1 |𝑠 𝑖 | > 𝐺 ∑︁ > 𝑖 =1 > |𝑠 𝑖 | ∑︁ > 𝑡 =1 min (𝑟 𝑖,𝑡 (𝜃 ) b𝐴 𝑖,𝑡 , clip (𝑟 𝑖,𝑡 (𝜃 ), 1 − 𝜀 low , 1 + 𝜀 high ) b𝐴 𝑖,𝑡 ) − 𝛽𝐷 KL (𝜋 𝜃 ∥𝜋 ref ) s.t. 0 < |{ 𝑠 𝑖 : is_equivalent (𝑎, 𝑠 𝑖 )}| < 𝐺 where 𝑟 𝑖,𝑡 (𝜃 ) = 𝜋 𝜃 (𝑠 𝑖,𝑡 |𝑞,𝑠 𝑖,<𝑡 ) > 𝜋 𝜃 old (𝑠 𝑖,𝑡 |𝑞,𝑠 𝑖,<𝑡 ) is the token-level weight, b𝐴 𝑖,𝑡 = 𝑅 𝑖 −mean ({𝑅 𝑖 }𝐺 𝑖 =1) > std ({𝑅 𝑖 }𝐺 𝑖 =1) is the advantage at the token level, and is_equivalent step filters out the prompts with accuracy equal to 1 and 0. The hyperparameters 𝜀 low and 𝜀 high control the clipping range for importance weights, while 𝛽 regulates the KL divergence penalty to stabilize training. Additional Designs. Beyond standard RL training, we implement two enhancements to stabilize tool-augmented judgment: (i) Error Message Processing. We truncate the outputs from Interpreter I to only the final error line to avoid excessive context length while preserving useful feedback in 𝑠 𝑘 ; (ii) Sandbox Output Masking. Since execution results 𝑜 𝑖 = I ( 𝑐 𝑖 ) may cause the model to overfit by memorizing outputs, we mask 𝑜 𝑖 during loss computation, following Jin et al. (2025); Li et al. (2025b). This prevents reliance on exact strings and improves training stability. Reward Designs. To effectively facilitate multi-turn RL with code execution, we design a structured covering three aspects, described as follows: (i) Correctness Reward 𝑅 𝑐 : This component measures whether the judge’s prediction aligns with the reference preference label. Let 𝑥 denote the prompt, Y = { 𝑦 1, . . . , 𝑦 𝑛 } the candidate responses, and 𝑙 the ground-truth preferred response. The reward is defined as: 𝑅 𝑐 =  𝕀 𝑠 𝜃 (𝑥, 𝑦 pos ) > 𝑠 𝜃 (𝑥, 𝑦 neg ), for pointwise evaluation , 𝕀 𝐽 𝜃 (𝑥, Y) = 𝑙 , for pairwise or listwise evaluation , 0, otherwise , (2) where 𝕀 (·) is the indicator function, 𝑠 𝜃 (𝑥, 𝑦 ) denotes the judge’s scoring function, and 𝐽 𝜃 (𝑥, Y) is the predicted best response under the judge’s policy. Intuitively, 𝑅 𝑐 = 1 if the judge’s decision matches the ground-truth preference, and 𝑅 𝑐 = 0 otherwise. > 5Incentivizing Agentic Reasoning in LLM Judges via Tool-Integrated Reinforcement Learning (ii) Format Reward 𝑅 𝑓 : To ensure reliability, the judge is required to strictly follow a predefined structured output format. Specifically, prediction scores must be enclosed within and tags, the preference label must be wrapped in and tags, and all code segments must be enclosed using ``` python and ``` . In addition, to accommodate both reasoning and non-reasoning tasks and discourage unnecessary tool calls , we introduce a heuristic: for safety and general helpfulness prompts, a positive format reward is granted only if the model produces a valid output without invoking tools. Formally, 𝑅 𝑓 = 1 if the output satisfies all formatting constraints (and the no-tool heuristic when applicable), and 𝑅 𝑓 = 0 otherwise. (iii) Tool-Specific Reward 𝑅 𝑡 : We encourage accurate and efficient tool use by penalizing errors or excessive executions (Wang et al., 2025a). We set the max number of tool calls per trajectory to 3, and set 𝑅 𝑡 = 1 only when code blocks 𝑐 𝑖 are error-free and within the call budget; otherwise 𝑅 𝑡 = 0.The final reward 𝑅 is defined as a combination of correctness, format, and tool-specific rewards and assigns full credit only when correctness, format, and tool-use are all satisfied: 𝑅 = 𝑅 𝑐 × 0.1 + 0.9 𝕀 [𝑅 𝑡 = 1 ∧ 𝑅 𝑓 = 1]. (3) 4.3. Training Strategies for RL Directly applying RL often leads to suboptimal outcomes, as the base model lacks sufficient reasoning and tool-use capability. To address this, we design two cold-start strategies for training TIR-Judge . Distillation from Teacher Models ( TIR-Judge -Distill). We leverage a stronger teacher, Gemini-2.5 - Flash with code execution (Comanici et al., 2025), to generate high-quality trajectories via rejection sampling. For each user prompt 𝑥 and corresponding Y, we collect a trajectory 𝑠 and a final prediction 𝑎 as (𝑥, Y, 𝑠, 𝑎 ) ∼ 𝐽 . Only trajectories that produce correct answers are retained, yielding a dataset TSFT = {( 𝑥, Y, 𝑠, 𝑎 ) | 𝑅 (𝑠, 𝑎, 𝑙 ) = 1}. Then the student judge is trained via supervised fine-tuning (SFT) with objective LSFT = −𝔼 (𝑥,𝜏 )∼T SFT " | 𝑦 |∑︁ > 𝑖 =1 log 𝑓 𝜃 (𝜏 𝑖 | 𝜏 <𝑖 , 𝑥 ) # , where 𝜏 = (𝑠, 𝑎 ) is the target trajectory with reasoning and code steps. As in RL training, interpreter feedback tokens are masked to prevent learning on execution results. In total, we collect about 10k tool-integrated trajectories for SFT, which serve as the initialization before reinforcement learning. Iterative Training without Distillation ( TIR-Judge -Zero). Beyond teacher distillation, we investi-gate whether tool-augmented judges can improve purely through self-bootstrapping (Huang et al., 2023; Singh et al., 2024; Xiong et al., 2025; Yuan et al., 2024; Zelikman et al., 2022). The process alternates between RL, rejection sampling, and supervised fine-tuning. Starting from the initial model 𝜋 𝜃 0 , we first obtain the checkpoint 𝜋 𝜃 1 via direct RL on training data as 𝜋 𝜃 1 ← RL (𝜋 𝜃 0 ) (Sec. 4.2). Then, for each prompt 𝑥 , we sample multiple trajectories from 𝜋 𝜃 1 as {𝑠 𝑖 }𝐺 𝑖 =1 ∼ 𝜋 𝜃 𝑡 (· | 𝑥 ) (𝐺 = 4 in our study), where each trajectory contains reasoning, code, and execution results: 𝑠 𝑖 = {𝑟 1, 𝑐 1, 𝑜 1, . . . , 𝑟 𝑘 , 𝑐 𝑘 , 𝑜 𝑘 }. We retain only valid trajectories that (i) produce the correct answer 𝑙 , (ii) satisfy the output format, and (iii) execute without interpreter errors as T𝑡 = {( 𝑥, 𝑠, 𝑎 ) | 𝑅 (𝑠, 𝑎, 𝑙 ) = 1}. To promote efficiency, for each prompt we further keep only one trajectory, preferring the shortest response or the one with the fewest tool calls. The dataset T𝑡 is then used for SFT, and the fine-tuned model initializes the next RL round. After each cycle, we select the best checkpoint based on held-out validation accuracy and repeat the RS → SFT → RL loop: T𝑡 +1 ← RS (𝜋 𝜃 𝑡 ), 𝜋 𝜃 𝑡 +1 ← SFT (𝜋 𝜃 𝑡 , T𝑡 +1), 𝜋 𝜃 𝑡 +1 ← RL (𝜋 𝜃 𝑡 +1 ). > 6Incentivizing Agentic Reasoning in LLM Judges via Tool-Integrated Reinforcement Learning This iterative process allows TIR-Judge -Zero to progressively bootstrap stronger reasoning and tool-use capabilities entirely from a initial model and facilitates self-improvement without distillation. 5. Experiments 5.1. Experiment Setups Evaluation Datasets. Following prior work (Chen et al., 2025b; Whitehouse et al., 2025), we focus on reasoning tasks , evaluating TIR-Judge on PPE Correctness (Frick et al., 2025). We additionally consider two more challenging datasets on judges: IFBench (Peng et al., 2025) for instruction-following and CodeJudgeBench (Jiang et al., 2025b) for code generation. All evaluations are conducted under both pointwise and pairwise settings to demonstrate the broader applicability of TIR-Judge . We also evaluate on general-domain judge benchmarks, where reasoning constitutes a subset, including RewardBench (Lambert et al., 2025), RM-Bench (Liu et al., 2025b) and JudgeBench (Tan et al., 2025) for pointwise/pairwise evaluation, and RewardBench 2 (Malik et al., 2025) for listwise evaluation. Implementation Details. We use Qwen3-8B and Qwen3-4B-Instruct-2507 (Team, 2025) as backbones, without enabling thinking mode, and implement training with Verl-Tool (Jiang et al., 2025a). For SFT, we train with batch size 64, learning rate 2e-6, context length 8192, for 1 epoch. For RL, we set the micro batchsize per gpu to 4, mini batchsize to 128 and number of rollout to 8. We set 𝜀 low = 0.2, 𝜀 high = 0.3, 𝛽 = 0.01 , max response length to 8192, learning rate 1e-6 and train for 2 epochs. 5% of the prompts from each task were hold-out for validation. The experiments are run with 8 NVIDIA H100 80G GPUs. For data collection in Sec. 4.1, we generate 2 rollouts for each model with 𝑡 = 0.9, 𝑝 = 0.95 . No external feedback (e.g., GPT annotations) is used. Baselines. We consider the following baselines: (i) Off-the-shelf LLM judges : GPT-4o (Hurst et al., 2024), GPT-o1-mini (Jaech et al., 2024), Deepseek-R1 (Guo et al., 2025a), Claude 3.5 (Anthropic, 2025), Gemini-2.5-Flash (Comanici et al., 2025), Qwen-3 (Team, 2025); (ii) Standard Reward Models : Armo-RM (Wang et al., 2024a), Skywork-Reward-Gemma-2 (Liu et al., 2024), Deepseek-BTRM (Liu et al., 2025c); (iii) Text-based Judges trained with RL : Deepseek-GRM (Liu et al., 2025c), J1 (Whitehouse et al., 2025), RM-R1 (Chen et al., 2025b), RRM (Guo et al., 2025b) and Think-RM (Hong et al., 2025); (iv) Tool-augmented Judges : Gemini-2.5-Flash-Tool (Comanici et al., 2025), AgentRM (Peng et al., 2025) 2, and Qwen-3 (Team, 2025) with the same tool as TIR-Judge . 5.2. Main Experiment Results Experiments for Pointwise/Pairwise Judging tasks. Table 1 shows the main results of TIR-Judge on six judge benchmarks. The per-task accuracy on several benchmark is deferred to Table 4. From the results, we have the following key observations: (i) TIR-Judge achieves strong judging accuracy compared to baselines. Notably, on the PPE benchmark, TIR-Judge outperforms baselines with simi-lar sizes by 4.8%-9.9% for pointwise judging and 4.5%-8.8% for pairwise judging. It also achieves com-petitive or even better performance on other benchmarks with baselines having more parameters and trained with more data. For example, TIR-Judge achieves similar accuracy on PPE and RewardBench compared to RRM-32B despite having only 1/4-1/8 of its parameters. (ii) RL is critical for boosting tool-use capability for judges : Simply augmenting Qwen-3 models with code execution yields negli-gible (<1%) or even negative gains. In contrast, RL produces substantial improvements, showing that base checkpoints lack robust code generation ability and that RL is essential for unlocking tool-use capa-bility. Moreover, RL confers strong generalization: although most IF data is verifiable, TIR-Judge also performs well on IFBench, which contains many non-verifiable constraints. (iii) Iterative RL is surpris- > 2For fairness, we use Qwen-3 as the backbone for AgentRM. AgentRM also leverages Armo-RM to assist judgment. > 7 Incentivizing Agentic Reasoning in LLM Judges via Tool-Integrated Reinforcement Learning Table 1 | Main results on six benchmarks. † indicates results reported from the original papers. CJBench, RWBench, and JGBench denote CodeJudgeBench, RewardBench, and JudgeBench. “Distill?” specifies whether the model relies on additional judge data distilled from teacher models. Bold highlights the overall best accuracy, while blue and red mark the best results within our direct comparisons for pointwise and pairwise settings, respectively. Baselines |Train| Distill? PPE Correctness IFBench CJBench RWBench RMBench JGBench MMLU-P MATH GPQA MBPP-P IFEval Avg. LLM-as-a-Judge (Pairwise Evaluation unless specified) Qwen3-4B-Instruct (Pointwise) – – 64.3 83.1 38.0 62.4 55.2 60.6 56.2 16.6 76.5 66.9 50.8 Qwen3-8B (Pointwise) – – 68.7 64.2 56.5 58.9 57.4 61.1 55.9 54.9 79.2 69.3 64.9 Gemini-2.5-Flash (Pointwise) – – 56.5 79.5 46.4 63.0 63.9 61.9 51.6 53.3 80.7 70.8 66.9 GPT-4o † – – – – – – – 57.6 61.3 – 86.7 72.5 56.6 GPT-o1-mini † – – – – – – – 71.3 70.1 – 87.1 – 65.7 DeepSeek-R1-671B † – – – – – – – 76.5 68.0 – 90.6 – 73.1 Claude 3.5 † – – 81.0 86.0 63.0 54.0 58.0 68.4 – 58.3 84.2 61.0 64.3 Qwen3-4B-Instruct (Pairwise) – – 63.9 83.1 35.0 59.7 60.7 60.4 62.2 34.5 86.0 75.3 63.9 Qwen3-8B (Pairwise) – – 73.8 80.2 57.3 57.8 58.4 65.5 61.3 60.8 87.0 77.9 67.5 Gemini-2.5-Flash (Pairwise) – – 68.8 85.5 58.1 86.5 75.0 74.8 69.3 66.5 93.4 81.9 75.4 Scalar Reward Models (Pointwise) Armo-RM-8B † 1000k ✗ 66.0 71.0 57.0 54.0 58.0 61.2 62.9 – 90.3 67.7 –Skywork-Gemma-2-27B † 80k ✗ 55.0 46.2 44.7 69.1 58.3 54.7 63.2 – 93.8 67.3 –Deepseek-BTRM-27B † 237k ✗ 68.8 73.2 56.8 68.8 66.0 66.7 – – 81.7 – – Text-based Reasoning Judges (Pairwise Evaluation unless specified) Deepseek-GRM-27B † 237k ✗ 64.8 68.8 55.6 50.1 59.8 59.8 – – 86.1 – –J1-8B (Pairwise) † 22k ✗ 65.6 70.0 53.2 53.1 54.0 59.2 – – 85.7 73.4 42.0 J1-8B (Pointwise) † 22k ✗ – – – – – 58.5 – – – – –RRM-7B 420k ✗ 66.5 88.0 57.9 61.2 53.6 65.4 60.1 63.4 82.2 70.4 67.0 RM-R1-Deepseek-Distill-7B 73k ✓ 67.3 91.2 62.6 60.5 53.0 66.9 56.6 63.2 80.1 72.4 67.7 RM-R1-Instruct-7B 73k ✓ 64.1 74.5 60.7 57.3 57.8 62.9 59.0 57.5 85.2 70.2 60.3 Think-RM 7B 10k ✓ 66.5 78.3 55.6 58.1 63.9 64.5 57.4 54.6 86.0 73.9 64.6 Tool-augmented Judges Qwen3-4B-Tool (Pointwise) – – 64.6 81.6 38.3 61.0 49.8 59.1 44.1 18.0 78.4 72.1 56.6 Qwen3-8B-Tool (Pointwise) – – 67.0 72.4 54.0 56.0 34.0 56.7 27.1 45.9 78.0 67.9 59.4 Gemini-2.5-Flash-Tool (Pointwise) – – 68.2 86.0 48.9 58.7 73.5 67.1 53.0 47.9 81.3 71.2 66.5 TIR-Judge -Distill 4B (Pointwise) 26k ✓ 58.7 81.9 45.8 64.1 78.9 65.9 65.8 59.9 76.6 71.9 66.7 TIR-Judge -Zero 4B (Pointwise) 26k ✗ 62.5 87.3 54.7 64.8 79.8 69.8 65.9 61.5 77.3 72.8 70.4 TIR-Judge -Distill 8B (Pointwise) 26k ✓ 70.9 88.1 52.3 61.0 83.0 71.0 68.4 61.9 81.0 76.7 68.2 TIR-Judge -Zero 8B (Pointwise) 26k ✗ 67.8 88.0 53.2 64.7 77.8 70.3 66.8 60.8 81.4 76.3 67.5 AgentRM 8B + 8B (Pairwise) – – 64.6 76.0 52.8 61.7 73.0 65.6 67.0 59.2 87.7 69.7 59.4 Qwen3-4B-Tool (Pairwise) – – 63.5 83.3 35.9 58.9 62.3 60.8 59.2 29.2 85.2 75.7 63.0 Qwen3-8B-Tool (Pairwise) – – 72.0 85.2 56.0 54.3 60.8 65.7 52.5 54.9 86.2 77.3 65.9 Gemini-2.5-Flash-Tool (Pairwise) – – 73.1 87.5 60.2 85.2 84.0 78.0 68.5 66.3 90.1 80.9 74.6 TIR-Judge -Distill 4B (Pairwise) 26k ✓ 69.0 88.7 54.8 60.6 83.6 71.3 73.7 69.8 87.7 78.0 70.5 TIR-Judge -Zero 4B (Pairwise) 26k ✗ 75.0 93.3 61.7 67.3 84.5 76.3 70.3 70.8 86.7 80.8 73.7 TIR-Judge -Distill 8B (Pairwise) 26k ✓ 72.2 90.4 53.8 63.2 85.7 73.0 74.3 70.0 87.9 82.2 72.6 TIR-Judge -Zero 8B (Pairwise) 26k ✗ 76.6 94.0 58.5 68.8 80.8 75.7 68.9 69.3 89.1 83.7 72.0 For Reference: Text-based Reasoning Judge Baselines with >10B Parameters (Pairwise Evaluation) J1 70B † 22k ✗ 79.0 86.0 65.9 66.0 67.3 72.8 – – 93.3 82.7 60.0 RRM 32B 420k ✗ 80.5 94.3 68.4 72.8 60.2 75.3 60.8 76.3 91.2 85.4 76.0 RM-R1-Deepseek-Distill-14B 73k ✓ 78.8 94.5 63.3 70.5 63.0 74.0 58.6 65.5 88.9 81.5 76.2 RM-R1-Deepseek-Distill-32B 73k ✓ 79.8 95.4 65.2 74.6 63.3 75.6 60.4 65.8 90.9 83.9 78.4 ingly effective to reduce the need for distillation : Comparing TIR-Judge -Zero with TIR-Judge -Distill, we find that TIR-Judge -Zero delivers comparable or better performance, outperforming the distilled variant on 4/6 benchmarks (pointwise) and 3/6 benchmarks (pairwise). This demon-strates iterative RL as an efficient alternative when supervision from frontier models are unavailable. Table 2 | Results on 5 tasks in RewardBench2, sorted by average performance. Datasets IF Math Fact Focus Safety Avg. Claude-Opus-4 41.9 74.9 82.7 86.2 89.5 76.5 Gemini-2.5-flash-Preview 55.3 81.1 65.7 86.7 90.9 75.9 TIR-Judge -Zero 8B 45.6 84.1 64.8 89.5 82.7 73.4 TIR-Judge -Distill 8B 58.1 72.7 63.8 81.4 82.0 71.6 GPT-4.1 39.7 65.2 82.9 73.4 87.3 69.7 Claude-Sonnet-4 35.9 70.5 76.1 76.0 89.1 69.5 TIR-Judge -Zero 4B 47.5 86.4 59.3 85.2 62.9 68.3 TIR-Judge -Distill 4B 55.0 78.1 55.8 75.0 73.1 67.3 GPT-4.1-mini 41.2 72.1 60.8 73.5 72.6 65.7 GPT-4o 33.1 62.3 56.8 72.9 86.2 64.9 Experiments on Listwise Judging tasks. We further evaluate TIR-Judge on Reward-Bench2 (Malik et al., 2025) under listwise judge setting, where the input contains one chosen and multiple rejected responses. As shown in Ta-ble 2, TIR-Judge achieves strong performance, matching 96% performance of Claude-Opus-4, the current best model on the leaderboard, de-spite being 8B parameter only. The advantage is 8Incentivizing Agentic Reasoning in LLM Judges via Tool-Integrated Reinforcement Learning PPE-IFEval PPE-MBPP-Plus PPE-MATH PPE-MMLU-Pro PPE-GPQA RMBench-Chat RMBench-Safety > 25 > 50 > 75 > Performance > TIR-Judge-Zero 8B w/ only IF w/ only Coding w/ only Reasoning w/ only Helpfulness & Safety Figure 3 | The effect of different data mixture used in RL training of TIR-Judge -Zero. PPE-IFEval > PPE-MBPP-Plus > PPE-MATH > PPE-MMLU-Pro > PPE-GPQA > RMBench-Chat > RMBench-Safety > 25 > 50 > 75 > Performance > Direct RL ( 1) w/ Tool Direct RL ( 1) w/o Tool (a) Tool Use v.s. Text-only Judges AIME 2024 AIME 2025 BigCodeBench IFEval > 25 > 50 > 75 > Best of N Avg. Accuracy Pass@1 > MV@16 > RRM-7B (Pairwise) > TIR-Judge-Zero-4B Best > TIR-Judge-Zero-8B Best (b) Best-of-N Inference Figure 4 | Experimental results comparing tool-augmented judges against text-only judges under the same training data and settings, as well as the best-of-𝑁 inference performance. more notable on tasks such as instruction following and mathematical reasoning, where TIR-Judge ’s integration of code execution provides a clear gain. 5.3. Additional Studies Diverse Data Mixture is essential for RL. We study the impact of task composition in RL in Figure 3. Training exclusively on chat or reasoning tasks leads to poor transfer across subtasks, largely because the scarcity of tool-use prompts prevents the model from fully developing tool-use capabilities. In contrast, unifying tasks – both with and without tool use – into a single training pipeline leads to improved generalization. Tool Use vs. Text-Only. To rigorously evaluate the impact of tool integration, we conduct a controlled study in which code execution is disabled during RL while keeping the training data identical. As shown in Figure 4(a), tool-augmented models achieve consistently higher accuracy on reasoning and IF benchmarks, while text-only models perform slightly better on text-centric tasks such as Chat and Safety in RMBench. These comparisons highlight the strength of tool-augmented judges for reasoning, and further suggest that mixing prompts from both tool-use and non–tool-use settings maintains robust performance without sacrificing much on cases where tools are unnecessary. 0 5 10 15 Avg. # Prompts per Second 60 65 70 75 > Avg. PPE Performance > Qwen-3-4B > Qwen-3-8B > RRM-7B > RM-R1-distill 7B > Think-RM 7B > RRM-32B > RM-R1-distill 32B > TIR-Judge-Distill 4B > TIR-Judge-Zero 4B > TIR-Judge-Distill 8B > TIR-Judge-Zero 8B Figure 6 | Study on Inference Efficiency. Efficiency Studies. We further evaluate the ef-ficiency of TIR-Judge against several baselines in Figure 6. While TIR-Judge achieves higher accu-racy, incorporating external code execution tools in-troduces no additional inference-time overhead. In fact, TIR-Judge is more efficient than the baselines, benefiting from our SFT data construction strategy that favors trajectories with shorter reasoning and fewer tool calls during rejection sampling. Iterative RL progressively improves TIR-Judge -Zero. We evaluate TIR-Judge -Zero across training stages under the pairwise setting. As shown in Figure 5, we observe substantial gains after the first round > 9Incentivizing Agentic Reasoning in LLM Judges via Tool-Integrated Reinforcement Learning 4B 8B > 60 > 80 > Performance > PPE > 4B 8B > 50 > 60 > 70 > IFBench > 4B 8B > 40 > 60 > CodeJudgeBench > Base > TIR-Judge-Zero-RL-0 > TIR-Judge-Distill-SFT > TIR-Judge-Zero-RL-1 > TIR-Judge-Distill-RL > TIR-Judge-Zero-RL-2 > TIR-Judge-Zero-RS Figure 5 | Accuracy of TIR-Judge across different training stages. Base denotes the backbone model without additional training. TIR-Judge -Zero-RS is a variant used in Zelikman et al. (2022) that uses rejection sampling to construct high-quality trajectories for SFT (without RL). TIR-Judge -Zero-RL-0 refer to the judge with direct RL training, and TIR-Judge -Zero-RL-0 refer to the performance of TIR-Judge after 1, and 2 iterations of RS-SFT-RL cycles, respectively. of RL. These improvements arise from rejection sampling, which teaches the model to produce more format-correct and efficient tool use, thereby strengthening its reasoning capability. Additional RL iterations further boost accuracy as RL benefits from progressively higher-quality SFT data. In contrast, rejection-sampling fine-tuning yields modest gains, highlighting the necessity of online RL. 5.4. Best-of-N Evaluation on Policy Models We conduct parallel test-time compute scaling experiment to study whether TIR-Judge can improve the downstream performance of the policy model, where we conduct a study on reward-guided best-of-N inference over datasets from multiple domains including AIME-2024 , AIME-2025 , Big-CodeBench (Zhuo et al., 2025) and IFEval (Zhou et al., 2023). The detailed experimental setup is deferred to the Appendix E. Figure 4(b) presents the average accuracy of TIR-Judge over different LLM policy models compared to a strong baseline, RRM, across four datasets. We find that TIR-Judge consistently surpasses both Majority Voting (Self-Consistency; Wang et al. (2023)) and RRM by clear margins, demonstrating its effectiveness. The improvements are especially pronounced on challenging bench-marks: BigCodeBench, which involves complex code generation and diverse functions, and AIME, which consists of competition-level math problems. On these tasks, TIR-Judge achieves absolute gains of 3.9–6.7% over RRM. This justifies its ability to handle more challenging tasks in real-world applications. 5.5. Case Studies Table 3 presents an example from the IFEval subset of the PPE benchmark. TIR-Judge successfully generates correct Python functions to verify two responses and produces the correct pairwise judgment. In contrast, text-only judges struggle, as counting remains challenging and often leads to incorrect and hallucinated reasoning steps, which yield incorrect predictions. This highlights how tool integration enables TIR-Judge to overcome failure modes that remain difficult for text-only judges. 6. Conclusion In this work, we introduce TIR-Judge , the first tool-integrated framework for training LLM judges with end-to-end reinforcement learning. Different from prior works on text-only judges, TIR-Judge tightly couples reasoning with code execution to enable judges to perform precise verification and > 10 Incentivizing Agentic Reasoning in LLM Judges via Tool-Integrated Reinforcement Learning Table 3 | A case study from IFEval demonstrating the judgement generated by TIR-Judge -Zero 8B and Qwen3-8B. Red text stands for the incorrect reasoning step while blue text justify the ability of TIR-Judge in writing correct verification codes. Part of responses are omitted due to space limits. User Prompt Write a letter to your friend who recently moved away. Your response should be in English, and in all capital letters. The letter o should appear at least 40 times. Response A MY DEAREST FRIEND,IT HAS BEEN SO LONG SINCE YOU MOVED AWAY, AND I MISS YOU TERRIBLY. ...... I CAN’T WAIT TO HEAR FROM YOU AND TO CATCH UP. WITH LOTS OF LOVE, YOUR FRIEND Response B MY DEAR FRIEND,IT HAS BEEN SOME TIME SINCE YOU MOVED AWAY, ...... I WISH YOU NOTHING BUT THE BEST ON YOUR NEW ADVENTURE, FRIEND.remember M UPDATE WITH ALL THE DETAILS ABOUT YOUR LIFE THEREOOVER IN THE PRINTED PAGES. PLEASE SEND Oast THOUGHTS YOUR WAY, AS I KNOW YOURS ARE ALWAYS NEARLY ALWAYS WITH YOUR FRIEND FOR LIFE Judge Trajectory by TIR-Judge -Zero 8B To evaluate the responses, I will perform the following checks: 1. Ensure both responses are in all capital letters. 2. Count the number of times the letter "O" appears in each response. “‘python # Count occurrences of ’O’ (case-sensitive) count_o_a = response_a.count(’O’) count_o_b = response_b.count(’O’) # Check if responses are in all capital letters is_upper_a = response_a == response_a.upper() is_upper_b = response_b == response_b.upper() print(f"Response A - ’O’ count: count_o_a, All caps: is_upper_a") print(f"Response B - ’O’ count: count_o_b, All caps: is_upper_b") “‘ “‘output Response A - ’O’ count: 58, All caps: True Response B - ’O’ count: 60, All caps: False “‘ A Qwen3-8B Analysis of Response A: - Language: Entirely in English. ✓ - Capitalization: All text is in all capital letters. ✓ - Letter "O" Count: Let’s count the number of "O" letters in Response A (excluding punctuation and spaces for clarity): - "MY DEAREST FRIEND," → 1 "O" - "IT HAS BEEN SO LONG SINCE YOU MOVED AWAY, AND I MISS YOU TERRIBLY." → 2 "O"s ...... Total O’s in Response A: 1 + 2 + 3 + 3 + 4 + 1 + 2 + 2 + 2 + 2 + 2 + 1 + 1 = 23 → Does NOT meet the requirement of at least 40 "O" letters. ✗ Analysis of Response B: - Language: Entirely in English. ✓ - Capitalization: All text is in all capital letters. ✓ - Letter "O" Count: Let’s count the number of "O" letters in Response A (excluding punctuation and spaces for clarity): - "MY DEAR FRIEND," → 2 "O"s - "IT HAS BEEN SOME TIME SINCE YOU MOVED AWAY," → 1 "O" - "I HAVE TO ADMIT, I MISS OUR THOUGHTFUL CONVERSATIONS AND EVEN YOUR (ADMIT IT) TERRIBLE, WONDERFUL COOKING." → 4 "O"s ...... Total O’s in Response B: 2 + 1 + 4 + 1 + 4 + 1 + 7 = 20 → Does NOT meet the requirement of at least 40 "O" letters. ✗ Conclusion: Both responses fail to meet the instruction’s requirement that the letter "O" must appear at least 40 times. However, Response B contains more "O" letters (20) compared to Response A (23), which is still below the required threshold. Despite both responses not fully complying with the instruction, Response B is more closely aligned with the instruction, as it contains a higher number of "O" letters and is more detailed in content. B computation. To maximize the benefits of RL, we propose three key design choices: task diversification , flexible judgement , and iterative RL training . Experiments on seven benchmarks show that TIR-Judge outperforms strong reasoning judges by up to 6.4% (pointwise) and 7.7% (pairwise), and matches 96% of Claude-Opus-4’s listwise performance with only 8B parameters. TIR-Judge -Zero shows that pure RL can bootstrap tool-use without supervision, offering a scalable path toward self-improving judges. In future work, we aim to expand the range of tools and training tasks used in RL, and explore using TIR-Judge to enhance policy model training. 11 Incentivizing Agentic Reasoning in LLM Judges via Tool-Integrated Reinforcement Learning Acknowledgement We would like to thank Jing Nathan Yan, Zhengzhe Yang, Yuchen Zhuang from Google DeepMind for useful discussions. References Z. Ankner, M. Paul, B. Cui, J. D. Chang, and P. Ammanabrolu. Critique-out-loud reward models. arXiv preprint arXiv:2408.11791 , 2024. Anthropic. Introducing claude 3.5 sonnet. Technical report, 2025. F. Bai, Y. Min, B. Zhang, Z. Chen, W. X. Zhao, L. Fang, Z. Liu, Z. Wang, and J.-R. Wen. Towards effective code-integrated reasoning. arXiv preprint arXiv:2505.24480 , 2025. N. Chen, Z. Hu, Q. Zou, J. Wu, Q. Wang, B. Hooi, and B. He. Judgelrm: Large reasoning models as a judge. arXiv preprint arXiv:2504.00050 , 2025a. W. Chen, X. Ma, X. Wang, and W. W. Cohen. Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks. Transactions on Machine Learning Research , 2023. URL https://openreview.net/forum?id=YfZ4ZPt8zd .X. Chen, G. Li, Z. Wang, B. Jin, C. Qian, Y. Wang, H. Wang, Y. Zhang, D. Zhang, T. Zhang, et al. Rm-r1: Reward modeling as reasoning. arXiv preprint arXiv:2505.02387 , 2025b. G. Comanici, E. Bieber, M. Schaekermann, I. Pasupat, N. Sachdeva, I. Dhillon, M. Blistein, O. Ram, D. Zhang, E. Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261 , 2025. J. Dai, X. Pan, R. Sun, J. Ji, X. Xu, M. Liu, Y. Wang, and Y. Yang. Safe RLHF: Safe reinforcement learning from human feedback. In The Twelfth International Conference on Learning Representations ,2024. URL https://openreview.net/forum?id=TyFrPOKYXw .G. Dong, Y. Chen, X. Li, J. Jin, H. Qian, Y. Zhu, H. Mao, G. Zhou, Z. Dou, and J.-R. Wen. Tool-star: Empowering llm-brained multi-tool reasoner via reinforcement learning. arXiv preprint arXiv:2505.16410 , 2025. W. Du, H. Gong, Z. Ling, K. Liu, L. Shen, X. Yao, Y. Xu, D. Shi, Y. Yang, and J. Chen. Generalizable end-to-end tool-use rl with synthetic codegym. arXiv preprint arXiv:2509.17325 , 2025. J. Feng, S. Huang, X. Qu, G. Zhang, Y. Qin, B. Zhong, C. Jiang, J. Chi, and W. Zhong. Retool: Reinforcement learning for strategic tool use in llms. arXiv preprint arXiv:2504.11536 , 2025. A. Findeis, F. Weers, G. Yin, K. Ye, R. Pang, and T. Gunter. Can external validation tools improve annotation quality for LLM-as-a-judge? In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , 2025. E. Frick, T. Li, C. Chen, W.-L. Chiang, A. N. Angelopoulos, J. Jiao, B. Zhu, J. E. Gonzalez, and I. Stoica. How to evaluate reward models for RLHF. In The Thirteenth International Conference on Learning Representations , 2025. URL https://openreview.net/forum?id=cbttLtO94Q .L. Gao, A. Madaan, S. Zhou, U. Alon, P. Liu, Y. Yang, J. Callan, and G. Neubig. Pal: Program-aided language models. In International Conference on Machine Learning , pages 10764–10799. PMLR, 2023. > 12 Incentivizing Agentic Reasoning in LLM Judges via Tool-Integrated Reinforcement Learning D. Guo, D. Yang, H. Zhang, J. Song, P. Wang, Q. Zhu, R. Xu, R. Zhang, S. Ma, X. Bi, et al. Deepseek-r1 incentivizes reasoning in llms through reinforcement learning. Nature , 645(8081):633–638, 2025a. J. Guo, Z. Chi, L. Dong, Q. Dong, X. Wu, S. Huang, and F. Wei. Reward reasoning model. arXiv preprint arXiv:2505.14674 , 2025b. D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt. Measuring mathematical problem solving with the MATH dataset. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2) , 2021. URL https://openreview.net/forum?id=7Bywt2mQsCe .I. Hong, C. Yu, L. Qiu, W. Yan, Z. Xu, H. Jiang, Q. Zhang, Q. Lu, X. Liu, C. Zhang, and T. Zhao. Think-rm: Enabling long-horizon reasoning in generative reward models. arXiv preprint arXiv:2505.16265 ,2025. A. Huang, A. Block, Q. Liu, N. Jiang, A. Krishnamurthy, and D. J. Foster. Is best-of-n the best of them? coverage, scaling, and optimality in inference-time alignment. In Forty-second International Confer-ence on Machine Learning , 2025a. URL https://openreview.net/forum?id=QnjfkhrbYK .J. Huang, S. Gu, L. Hou, Y. Wu, X. Wang, H. Yu, and J. Han. Large language models can self-improve. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages 1051–1068, Dec. 2023. X. Huang, G. Franke, Z. Yang, J. Bai, W. Bai, J. Bi, Z. Ding, Y. Duan, C. Fan, W. Fan, et al. Loong: Synthesize long chain-of-thoughts at scale through verifiers. arXiv preprint arXiv:2509.03059 ,2025b. A. Hurst, A. Lerer, A. P. Goucher, A. Perelman, A. Ramesh, A. Clark, A. Ostrow, A. Welihinda, A. Hayes, A. Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276 , 2024. A. Jaech, A. Kalai, A. Lerer, A. Richardson, A. El-Kishky, A. Low, A. Helyar, A. Madry, A. Beutel, A. Carney, et al. Openai o1 system card. arXiv preprint arXiv:2412.16720 , 2024. D. Jiang, Y. Lu, Z. Li, Z. Lyu, P. Nie, H. Wang, A. Su, H. Chen, K. Zou, C. Du, et al. Verltool: Towards holistic agentic reinforcement learning with tool use. arXiv preprint arXiv:2509.01055 , 2025a. H. Jiang, Y. Chen, Y. Cao, H.-y. Lee, and R. T. Tan. Codejudgebench: Benchmarking llm-as-a-judge for coding tasks. arXiv preprint arXiv:2507.10535 , 2025b. B. Jin, H. Zeng, Z. Yue, J. Yoon, S. O. Arik, D. Wang, H. Zamani, and J. Han. Search-r1: Training LLMs to reason and leverage search engines with reinforcement learning. In Second Conference on Language Modeling , 2025. URL https://openreview.net/forum?id=Rwhi91ideu .N. Lambert, J. Morrison, V. Pyatkin, S. Huang, H. Ivison, F. Brahman, L. J. V. Miranda, A. Liu, N. Dziri, S. Lyu, et al. Tulu 3: Pushing frontiers in open language model post-training. arXiv preprint arXiv:2411.15124 , 2024. N. Lambert, V. Pyatkin, J. Morrison, L. Miranda, B. Y. Lin, K. Chandu, N. Dziri, S. Kumar, T. Zick, Y. Choi, N. A. Smith, and H. Hajishirzi. RewardBench: Evaluating reward models for language modeling. In Findings of the Association for Computational Linguistics: NAACL 2025 , pages 1755–1797, 2025. J. Li, S. Sun, W. Yuan, R.-Z. Fan, H. Zhao, and P. Liu. Generative judge for evaluating alignment. In The Twelfth International Conference on Learning Representations , 2024a. URL https://openreview. net/forum?id=gtkFw6sZGS . > 13 Incentivizing Agentic Reasoning in LLM Judges via Tool-Integrated Reinforcement Learning L. Li, Y. Chai, S. Wang, Y. Sun, H. Tian, N. Zhang, and H. Wu. Tool-augmented reward modeling. In The Twelfth International Conference on Learning Representations , 2024b. URL https://openreview. net/forum?id=d94x0gWTUX .X. Li, X. Li, S. Hu, Y. Guo, and W. Zhang. Verifybench: A systematic benchmark for evaluating reasoning verifiers across domains. arXiv preprint arXiv:2507.09884 , 2025a. X. Li, H. Zou, and P. Liu. Torl: Scaling tool-integrated rl. arXiv preprint arXiv:2503.23383 , 2025b. H. Lin and Z. Xu. Understanding tool-integrated reasoning. arXiv preprint arXiv:2508.19201 , 2025. C. Y. Liu, L. Zeng, J. Liu, R. Yan, J. He, C. Wang, S. Yan, Y. Liu, and Y. Zhou. Skywork-reward: Bag of tricks for reward modeling in llms. arXiv preprint arXiv:2410.18451 , 2024. T. Liu, R. Xu, T. Yu, I. Hong, C. Yang, T. Zhao, and H. Wang. Openrubrics: Towards scalable synthetic rubric generation for reward modeling and llm alignment. arXiv preprint arXiv:2510.07743 , 2025a. Y. Liu, Z. Yao, R. Min, Y. Cao, L. Hou, and J. Li. RM-bench: Benchmarking reward models of language models with subtlety and style. In The Thirteenth International Conference on Learning Representations , 2025b. URL https://openreview.net/forum?id=QEHrmQPBdd .Z. Liu, P. Wang, R. Xu, S. Ma, C. Ruan, P. Li, Y. Liu, and Y. Wu. Inference-time scaling for generalist reward modeling. arXiv preprint arXiv:2504.02495 , 2025c. X. Ma, Q. Liu, D. Jiang, G. Zhang, Z. Ma, and W. Chen. General-reasoner: Advancing llm reasoning across all domains. arXiv preprint arXiv:2505.14652 , 2025a. Z. Ma, X. Zhang, J. Zhang, J. Yu, S. Luo, and J. Tang. Dynamic scaling of unit tests for code reward modeling. arXiv preprint arXiv:2501.01054 , 2025b. S. Malik, V. Pyatkin, S. Land, J. Morrison, N. A. Smith, H. Hajishirzi, and N. Lambert. Rewardbench 2: Advancing reward model evaluation. arXiv preprint arXiv:2506.01937 , 2025. S. I. Mirzadeh, K. Alizadeh, H. Shahrokhi, O. Tuzel, S. Bengio, and M. Farajtabar. GSM-symbolic: Un-derstanding the limitations of mathematical reasoning in large language models. In The Thirteenth International Conference on Learning Representations , 2025. URL https://openreview.net/ forum?id=AjXkRZIvjB .N. Muennighoff, Z. Yang, W. Shi, X. L. Li, L. Fei-Fei, H. Hajishirzi, L. Zettlemoyer, P. Liang, E. Candès, and T. Hashimoto. s1: Simple test-time scaling. arXiv preprint arXiv:2501.19393 , 2025. Y. Oren, N. Meister, N. S. Chatterji, F. Ladhak, and T. Hashimoto. Proving test set contamination in black-box language models. In The Twelfth International Conference on Learning Representations ,2024. URL https://openreview.net/forum?id=KS8mIvetg2 .H. Peng, Y. Qi, X. Wang, Z. Yao, B. Xu, L. Hou, and J. Li. Agentic reward modeling: Integrating human preferences with verifiable correctness signals for reliable reward systems. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , 2025. Z. Qi, X. Liu, I. L. Iong, H. Lai, X. Sun, J. Sun, X. Yang, Y. Yang, S. Yao, W. Xu, J. Tang, and Y. Dong. WebRL: Training LLM web agents via self-evolving online curriculum reinforcement learning. In The Thirteenth International Conference on Learning Representations , 2025. URL https://openreview.net/forum?id=oVKEAFjEqv . > 14 Incentivizing Agentic Reasoning in LLM Judges via Tool-Integrated Reinforcement Learning S. Saha, X. Li, M. Ghazvininejad, J. E. Weston, and T. Wang. Learning to plan & reason for evaluation with thinking-LLM-as-a-judge. In Forty-second International Conference on Machine Learning , 2025. URL https://openreview.net/forum?id=PNRznmmWP7 .Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. Li, Y. Wu, et al. Deepseek-math: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300 , 2024. J. Shen, R. Xu, Y. Jun, Z. Qin, T. Liu, C. Yang, Y. Liang, S. Baumgartner, and M. Bendersky. Boosting reward model with preference-conditional multi-aspect synthetic data generation. arXiv preprint arXiv:2407.16008 , 2024. A. Singh, J. D. Co-Reyes, R. Agarwal, A. Anand, P. Patil, X. Garcia, P. J. Liu, J. Harrison, et al. Beyond human data: Scaling self-training for problem-solving with language models. Transactions on Machine Learning Research , 2024. URL https://openreview.net/forum?id=lNAyUngGFK .H. Song, J. Jiang, Y. Min, J. Chen, Z. Chen, W. X. Zhao, L. Fang, and J.-R. Wen. R1-searcher: Incen-tivizing the search capability in llms via reinforcement learning. arXiv preprint arXiv:2503.05592 ,2025. S. Tan, S. Zhuang, K. Montgomery, W. Y. Tang, A. Cuadron, C. Wang, R. Popa, and I. Stoica. Judgebench: A benchmark for evaluating LLM-based judges. In The Thirteenth International Conference on Learning Representations , 2025. URL https://openreview.net/forum?id=G0dksFayVq .G. Team, M. Riviere, S. Pathak, P. G. Sessa, C. Hardin, S. Bhupatiraju, L. Hussenot, T. Mesnard, B. Shahriari, A. Ramé, et al. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118 , 2024. G. Team, A. Kamath, J. Ferret, S. Pathak, N. Vieillard, R. Merhej, S. Perrin, T. Matejovicova, A. Ramé, M. Rivière, et al. Gemma 3 technical report. arXiv preprint arXiv:2503.19786 , 2025. Q. Team. Qwen3 technical report, 2025. URL https://arxiv.org/abs/2505.09388 .H. Wang, W. Xiong, T. Xie, H. Zhao, and T. Zhang. Interpretable preferences via multi-objective reward modeling and mixture-of-experts. In Findings of the Association for Computational Linguistics: EMNLP 2024 , pages 10582–10592, Nov. 2024a. H. Wang, C. Qian, W. Zhong, X. Chen, J. Qiu, S. Huang, B. Jin, M. Wang, K.-F. Wong, and H. Ji. Acting less is reasoning more! teaching model to act efficiently. arXiv preprint arXiv:2504.14870 , 2025a. T. Wang, I. Kulikov, O. Golovneva, P. Yu, W. Yuan, J. Dwivedi-Yu, R. Y. Pang, M. Fazel-Zarandi, J. Weston, and X. Li. Self-taught evaluators. arXiv preprint arXiv:2408.02666 , 2024b. X. Wang, J. Wei, D. Schuurmans, Q. V. Le, E. H. Chi, S. Narang, A. Chowdhery, and D. Zhou. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations , 2023. URL https://openreview.net/forum?id= 1PL1NIMMrw .Z. Wang, J. Zeng, O. Delalleau, H.-C. Shin, F. Soares, A. Bukharin, E. Evans, Y. Dong, and O. Kuchaiev. Helpsteer3-preference: Open human-annotated preference data across diverse tasks and languages. arXiv preprint arXiv:2505.11475 , 2025b. C. Whitehouse, T. Wang, P. Yu, X. Li, J. Weston, I. Kulikov, and S. Saha. J1: Incentivizing thinking in llm-as-a-judge via reinforcement learning. arXiv preprint arXiv:2505.10320 , 2025. > 15 Incentivizing Agentic Reasoning in LLM Judges via Tool-Integrated Reinforcement Learning Z. Wu, M. Yasunaga, A. Cohen, Y. Kim, A. Celikyilmaz, and M. Ghazvininejad. rewordbench: Bench-marking and improving the robustness of reward models with transformed inputs. arXiv preprint arXiv:2503.11751 , 2025. W. Xiong, C. Shi, J. Shen, A. Rosenberg, Z. Qin, D. Calandriello, M. Khalman, R. Joshi, B. Piot, M. Saleh, C. Jin, T. Zhang, and T. Liu. Building math agents with multi-turn iterative preference learning. In The Thirteenth International Conference on Learning Representations , 2025. URL https://openreview.net/forum?id=WjKea8bGFF .R. Xu, Y. Zhuang, Z. Dong, R. Wang, Y. Yu, J. C. Ho, L. Zhang, H. Wang, W. Shi, and C. Yang. Acesearcher: Bootstrapping reasoning and search for llms via reinforced self-play. In Proceedings of the Thirty-ninth Annual Conference on Neural Information Processing Systems , 2025. URL https: //openreview.net/forum?id=jSgCM0uZn3 .Z. Xue, L. Zheng, Q. Liu, Y. Li, X. Zheng, Z. Ma, and B. An. Simpletir: End-to-end reinforcement learning for multi-turn tool-integrated reasoning. arXiv preprint arXiv:2509.02479 , 2025. Y. Yan, J. Jiang, Z. Ren, Y. Li, X. Cai, Y. Liu, X. Xu, M. Zhang, J. Shao, Y. Shen, et al. Verify-bench: Benchmarking reference-based reward systems for large language models. arXiv preprint arXiv:2505.15801 , 2025. Q. Yu, Z. Zhang, R. Zhu, Y. Yuan, X. Zuo, Y. Yue, W. Dai, T. Fan, G. Liu, L. Liu, et al. Dapo: An open-source llm reinforcement learning system at scale. arXiv preprint arXiv:2503.14476 , 2025a. Y. Yu, Z. Chen, A. Zhang, L. Tan, C. Zhu, R. Y. Pang, Y. Qian, X. Wang, S. Gururangan, C. Zhang, M. Kambadur, D. Mahajan, and R. Hou. Self-generated critiques boost reward modeling for language models. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) , 2025b. L. Yuan, G. Cui, H. Wang, N. Ding, X. Wang, B. Shan, Z. Liu, J. Deng, H. Chen, R. Xie, Y. Lin, Z. Liu, B. Zhou, H. Peng, Z. Liu, and M. Sun. Advancing LLM reasoning generalists with preference trees. In The Thirteenth International Conference on Learning Representations , 2025. URL https: //openreview.net/forum?id=2ea5TNVR0c .W. Yuan, R. Y. Pang, K. Cho, X. Li, S. Sukhbaatar, J. Xu, and J. E. Weston. Self-rewarding language models. In Forty-first International Conference on Machine Learning , 2024. URL https://openreview.net/forum?id=0NphYCmgua .E. Zelikman, Y. Wu, J. Mu, and N. Goodman. STar: Bootstrapping reasoning with reasoning. In Advances in Neural Information Processing Systems , 2022. URL https://openreview.net/ forum?id=_3ELRdg2sgI .L. Zhang, A. Hosseini, H. Bansal, M. Kazemi, A. Kumar, and R. Agarwal. Generative verifiers: Reward modeling as next-token prediction. In The Thirteenth International Conference on Learning Representations , 2025. J. Zhao, R. Liu, K. Zhang, Z. Zhou, J. Gao, D. Li, J. Lyu, Z. Qian, B. Qi, X. Li, et al. Genprm: Scaling test-time compute of process reward models via generative reasoning. arXiv preprint arXiv:2504.00891 ,2025. J. Zhou, T. Lu, S. Mishra, S. Brahma, S. Basu, Y. Luan, D. Zhou, and L. Hou. Instruction-following evaluation for large language models. arXiv preprint arXiv:2311.07911 , 2023. Y. Zhou, A. Xu, P. Wang, C. Xiong, and S. Joty. Evaluating judges as evaluators: The jetts benchmark of llm-as-judges as test-time scaling evaluators. arXiv preprint arXiv:2504.15253 , 2025. > 16 Incentivizing Agentic Reasoning in LLM Judges via Tool-Integrated Reinforcement Learning Y. Zhuang, D. Jin, J. Chen, W. Shi, H. Wang, and C. Zhang. Workforceagent-r1: Incentivizing reasoning capability in llm-based web agents via reinforcement learning. arXiv preprint arXiv:2505.22942 ,2025. M. Zhuge, C. Zhao, D. R. Ashley, W. Wang, D. Khizbullin, Y. Xiong, Z. Liu, E. Chang, R. Krishnamoorthi, Y. Tian, et al. Agent-as-a-judge: Evaluate agents with agents. In Forty-second International Conference on Machine Learning , 2025. T. Y. Zhuo, V. M. Chien, J. Chim, H. Hu, W. Yu, R. Widyasari, I. N. B. Yusuf, H. Zhan, J. He, et al. Bigcodebench: Benchmarking code generation with diverse function calls and complex instructions. In The Thirteenth International Conference on Learning Representations , 2025. URL https:// openreview.net/forum?id=YrycTjllL0 . A. Prompt Template We provide the detailed prompt template used in TIR-Judge in the following. You are an expert judge for evaluating whether a model-generated response correctly answer the user’s instruction. Please carefully read both the Instruction and the Response below. Your goal is to assess whether the response accurately and completely follows the instruction, focusing strictly on correctness and fidelity. Do not consider writing style, tone, or fluency unless they affect instruction compliance. You may use Python code to assist your evaluation, by writing the code between the “‘python and “‘ tags following with “‘output. Each program between “‘python and “‘ tags are independent program. This is especially helpful for verifying complex requirements, such as checking capitalization, counting words, letters, keywords, or placeholders, checking the correctness of calculation steps for reasoning problems, as well as running test cases on the coding responses. Instruction: {prompt} Response: {response} Guidelines: In the end of your response, please give a score in range 1 to 10 in the format such as 10 to indicate the quality of the response. A score of 10 indicates that the response fully adheres to the instruction and provides a correct answer. A score of 1 indicates that the response is irrelevant, evasive, or fails to address key aspects of the instruction. Figure 7 | Prompt for response generation with tools on pointwise evaluation. > 17 Incentivizing Agentic Reasoning in LLM Judges via Tool-Integrated Reinforcement Learning You are an expert judge for evaluating whether a model-generated response correctly answer the user’s instruction. Please carefully read Instruction, Response A and Response B below. Your goal is to assess which response accurately and completely follows the instruction, focusing strictly on correctness and fidelity. Do not consider writing style, tone, or fluency unless they affect instruction compliance. You may use Python code to assist your evaluation, by writing the code between the “‘python and “‘ tags following with “‘output. Each program between “‘python and “‘ tags are independent program. This is especially helpful for verifying complex requirements, such as checking capitalization, counting words, letters, keywords, or placeholders, checking the correctness of calculation steps for reasoning problems, as well as running test cases on the coding responses. Instruction: {prompt} Response A: {response A} Response B: {response B} Guidelines: In the end of your response, please give a preference in the format such as A to indicate the better response. Figure 8 | Prompt for response generation with tools on pairwise evaluation. You are an expert judge for evaluating whether a model-generated response correctly answer the user’s instruction. Please carefully read Instruction and all responses below. Your goal is to assess which response accurately and completely follows the instruction, focusing strictly on correctness and fidelity. Do not consider writing style, tone, or fluency unless they affect instruction compliance. You may use Python code to assist your evaluation, by writing the code between the “‘python and “‘ tags following with “‘output. Each program between “‘python and “‘ tags are independent program. This is especially helpful for verifying complex requirements, such as checking capitalization, counting words, letters, keywords, or placeholders, checking the correctness of calculation steps for reasoning problems, as well as running test cases on the coding responses. Instruction: {prompt} Response A: {response A} Response B: {response B} Response C: {response C} Response D: {response D} ... Guidelines: In the end of your response, please give a preference in the format such as A to indicate the better response. Figure 9 | Prompt for response generation with tools on listwise evaluation. > 18 Incentivizing Agentic Reasoning in LLM Judges via Tool-Integrated Reinforcement Learning You are an expert judge for evaluating whether a model-generated response correctly answers the user’s instruction while being harmless. Please carefully read both the Instruction and the Response below. Your goal is to assess whether the response accurately and completely follows the instruction, focusing strictly on correctness and fidelity. Do not consider writing style, tone, or fluency unless they affect instruction compliance. Instruction: {prompt} Response: {response} Guidelines: Please first generate a textual reasoning step before making final predictions. In the end of your response, please give a score in range 1 to 10 in the format such as 10 to indicate the quality of the response. A score of 10 indicates that the response fully adheres to the instruction and provides a correct answer. A score of 1 indi-cates that the response is irrelevant, evasive, or fails to address key aspects of the instruction. Figure 10 | Prompt for response generation without tools on pointwise evaluation. You are an expert judge for evaluating whether a model-generated response correctly answers the user’s instruction while being harmless. Please carefully read the instructions and all responses below. Your goal is to assess which response accurately and completely follows the instruction, focusing strictly on correctness and fidelity. Instruction: {prompt} Response A: {response A} Response B: {response B} Guidelines: Please first generate a textual reasoning step before making final predictions. In the end of your response, please give a preference in the format such as A to indicate the better response. Figure 11 | Prompt for response generation without tools on pairwise evaluation. > 19 Incentivizing Agentic Reasoning in LLM Judges via Tool-Integrated Reinforcement Learning You are an expert judge for evaluating whether a model-generated response correctly answers the user’s instruction while being harmless. Please carefully read the instructions and all responses below. Your goal is to assess which response accurately and completely follows the instruction, focusing strictly on correctness and fidelity. Instruction: {prompt} Response A: {response A} Response B: {response B} Response C: {response C} Response D: {response D} ... Guidelines: Please first generate a textual reasoning step before making final predictions. In the end of your response, please give a preference in the format such as A to indicate the better response. Figure 12 | Prompt for response generation without tools on listwise evaluation. B. Full Performance on Several Benchmarks Table 4 shows the full results of TIR-Judge and key baselines on RewardBench, RMBench, and JudgeBench. Sometimes we observe that the performance of Gemini-2.5-flash declines when additional tools are introduced. This issue arises from a maximum-turn limit on tool calls: the model sometimes generates excessive tool invocations and, in certain cases, fails to terminate properly. C. Details on Training Data Composition Our training mixture spans reasoning, code evaluation, and safety alignment tasks for reinforcement learning. Table 5 summarizes dataset statistics across three supervision formats: pointwise, pairwise, and listwise. To ensure label reliability, we apply additional quality control. For HelpSteer3, we retain only examples where one response is explicitly annotated as better or significantly better, removing ambiguous preferences. For math and reasoning datasets with synthetic responses, we employ math-verify to automatically check the correctness of responses. For listwise data, we sample 3–5 negatives per instance and enforce that negatives yield different final answers from the positive, preventing trivial shortcut solutions. Finally, we address potential biases such as stylistic artifacts in evaluation datasets (Wu et al., 2025), reducing the risk of overfitting to surface-level patterns. D. Additional Implementation Details for Evaluation Implementation of different evaluation protocols. We list the implementation for different types of judging tasks as follows. • Pointwise: For pointwise evaluation, we follow the protocol of RewardBench2 (Malik et al., 2025), assigning partial credit of 0.5 when two responses are scored as a tie. Both TIR-Judge and pointwise baselines are evaluated under this rule. > 20 Incentivizing Agentic Reasoning in LLM Judges via Tool-Integrated Reinforcement Learning Table 4 | Detailed Per-task Experiment Results on RewardBench, RMBench, and JudgeBench. Baselines |Train| Distill? RewardBench RMBench JudgeBench Chat Chat-Hard Safety Reason Avg. Chat Math Code Safety Avg. Math Code Knowledge Reason Avg. LLM-as-a-Judge (Pairwise Evaluation unless specified) Qwen3-4B-Instruct (Pointwise) – – 81.0 73.9 77.0 74.3 76.5 67.8 82.1 38.4 79.2 66.9 65.5 35.4 58.2 37.6 50.8 Qwen3-8B (Pointwise) – – 79.1 74.2 79.9 83.4 79.2 64.1 74.7 56.6 81.7 69.3 63.6 64.6 64.4 66.5 64.9 Gemini-2.5-Flash (Pointwise) – – 71.8 77.0 93.0 80.9 80.7 59.5 77.3 56.0 90.6 70.8 71.4 73.8 61.0 70.4 66.9 GPT-4o † – – 96.1 76.1 86.6 88.1 86.7 67.2 67.5 63.6 91.7 72.5 75.0 59.5 50.7 54.1 56.6 GPT-o1-mini † – – 94.4 78.7 80.9 94.2 87.1 – – – – – 82.1 78.5 58.4 62.2 65.7 DeepSeek-R1-671B † – – 95.3 83.6 86.4 97.4 90.6 – – – – – 80.3 92.8 59.1 82.6 73.1 Claude 3.5 † – – 96.4 74.0 81.6 84.7 84.2 62.5 62.6 54.4 64.4 60.9 66.1 64.3 62.3 66.3 64.3 Qwen3-4B-Instruct (Pairwise) – – 93.0 80.2 80.1 90.6 86.0 75.2 81.7 67.3 77.1 75.3 69.1 70.7 56.2 70.1 63.9 Qwen3-8B (Pairwise) – – 94.1 79.0 85.8 89.2 87.0 78.6 82.9 61.6 88.6 77.9 75.0 66.3 65.4 67.0 67.5 Gemini-2.5-Flash (Pairwise) – – 95.0 87.9 97.5 92.7 93.4 78.5 75.6 80.0 93.7 81.9 85.7 88.1 70.1 72.4 75.4 Scalar Reward Models (Pointwise) Armo-RM-8B † 1000k ✗ 96.9 76.8 90.5 97.3 90.3 67.8 57.5 53.1 92.4 67.7 – – – – –Skywork-Gemma-2-27B † 80k ✗ 95.8 91.4 92.0 96.1 93.8 69.5 54.7 53.2 91.9 67.3 – – – – –Deepseek-BTRM-27B † 237k ✗ – – – – 81.7 – – – – – – – – – – Text-based Reasoning Judges (Pairwise Evaluation unless specified) Deepseek-GRM-27B † 237k ✗ 94.1 78.3 88.0 83.8 86.1 – – – – – – – – – –J1-8B (Pairwise) † 22k ✗ 92.9 80.3 85.6 83.9 85.7 – – – – 73.4 – – – – 42.0 J1-8B (Pointwise) † 22k ✗ – – – – 58.5 – – – – – – – – – –RRM-7B 420k ✗ 87.7 70.4 80.7 90.0 82.2 58.4 81.8 56.7 84.9 70.4 83.2 61.9 64.3 64.2 67.0 RM-R1-Deepseek-Distill-7B 73k ✓ 88.9 66.2 78.4 87.0 80.1 64.0 83.9 56.2 85.3 72.4 82.1 71.4 64.9 62.2 67.7 RM-R1-Instruct-7B 73k ✓ 94.1 74.6 85.2 86.7 85.2 66.6 67.0 54.6 92.6 70.2 76.8 54.8 56.4 59.2 60.3 Think-RM 7B 10k ✓ 94.4 77.9 85.2 86.4 86.0 69.3 76.0 56.5 93.7 73.9 67.9 42.9 67.5 67.3 64.6 Tool-augmented Judges Qwen3-4B-Tool (Pointwise) – – 81.0 74.8 77.2 80.5 78.4 68.2 82.4 58.6 79.3 72.1 63.6 42.7 57.8 57.7 56.6 Qwen3-8b-Tool (Pointwise) – – 77.6 75.3 80.7 78.5 78.0 63.4 71.2 55.9 81.0 67.9 59.1 57.3 56.2 65.5 59.4 Gemini-2.5-Flash Tool (Pointwise) – – 75.4 73.0 93.5 83.5 81.3 62.7 75.4 49.0 86.3 71.0 73.2 78.5 59.1 69.3 66.5 TIR-Judge -Distill 4B (Pointwise) 26k ✓ 79.7 66.5 82.9 77.2 76.6 61.8 81.2 56.7 87.9 71.9 71.8 70.7 60.8 71.7 66.7 TIR-Judge -Zero 4B (Pointwise) 26k ✗ 79.4 69.8 77.6 82.4 77.3 62.3 88.3 59.0 81.5 72.8 71.8 76.8 66.0 73.7 70.4 TIR-Judge -Distill 8B (Pointwise) 26k ✓ 78.3 73.9 84.9 87.0 81.0 65.6 85.8 65.7 89.7 76.7 78.1 75.5 64.4 65.5 68.2 TIR-Judge -Zero 8B (Pointwise) 26k ✗ 83.6 74.4 85.5 81.9 81.4 66.7 88.3 60.2 90.1 76.3 70.0 74.4 62.1 71.7 67.5 AgentRM 8B + 8B (Pairwise) – – 95.3 74.3 88.3 93.0 87.7 75.4 58.8 53.9 90.7 69.7 – – – – 59.4 Qwen3-4B-Tool (Pairwise) – – 92.7 78.7 80.9 88.5 85.2 79.1 83.2 63.2 77.5 75.7 72.7 58.5 60.8 62.9 63.0 Qwen3-8b-Tool (Pairwise) – – 93.3 78.5 86.2 86.8 86.2 77.5 82.4 60.8 88.3 77.3 78.2 61.0 64.1 63.9 65.9 Gemini-2.5-Flash Tool (Pairwise) – – 90.9 84.3 96.5 88.8 90.1 73.9 76.0 69.5 94.8 80.9 89.3 88.1 67.5 71.4 74.6 TIR-Judge -Distill 4B (Pairwise) 26k ✓ 95.0 75.2 88.9 91.6 87.7 71.6 86.3 61.4 92.9 78.0 81.8 82.9 60.8 74.2 70.6 TIR-Judge -Zero 4B (Pairwise) 26k ✗ 94.4 79.8 78.2 94.4 86.7 77.3 92.3 66.4 87.3 80.8 85.5 82.9 65.4 76.3 73.7 TIR-Judge -Distill 8B (Pairwise) 26k ✓ 92.2 75.6 89.0 94.8 87.9 78.6 89.0 67.7 93.5 82.2 90.2 76.4 68.0 68.0 72.6 TIR-Judge -Zero 8B (Pairwise) 26k ✗ 94.7 77.4 88.8 95.7 89.1 80.1 91.9 69.0 93.9 83.7 81.8 73.2 66.0 75.3 72.0 For Reference: Text-based Reasoning Judge Baselines with >10B Parameters (Pairwise Evaluation) J1 70B † 22k ✗ 96.1 90.1 91.9 94.9 93.3 – – – – 82.7 – – – – 60.0 RRM 32B 420k ✗ 94.7 81.1 90.7 98.3 91.2 73.9 91.8 74.8 95.3 85.4 87.5 85.7 68.8 76.5 76.0 RM-R1-Deepseek-Distill-14B 73k ✓ 91.3 79.4 89.3 95.5 88.9 71.8 90.5 69.5 94.1 81.5 89.2 88.0 70.1 73.4 76.2 RM-R1-Deepseek-Distill-32B 73k ✓ 95.3 80.3 91.1 96.8 90.9 74.2 91.8 74.1 95.4 83.9 92.8 82.3 72.7 77.5 78.4 21 Incentivizing Agentic Reasoning in LLM Judges via Tool-Integrated Reinforcement Learning Table 5 | Dataset statistics for pointwise, pairwise, and listwise data. > Dataset Domain Pointwise Pairwise Listwise Total > Tulu-3 Synthetic Pairs (Lambert et al., 2024) IF 1,500 1,500 263 3,263 MATH (Hendrycks et al., 2021) Math 1,000 1,000 254 2,254 dapo_bigmath (Yu et al., 2025a) Math 2,500 2,500 282 5,282 s1 (Muennighoff et al., 2025) Math 250 250 0500 UltraInteract (Yuan et al., 2025) Code 2,000 2,000 04,000 CodeRM (Ma et al., 2025b) Code 1,000 1,000 472 2,472 WebInstruct (Ma et al., 2025a) Reasoning 1,000 1,000 91 2,091 Loong (Huang et al., 2025b) Reasoning 700 700 99 1,499 HelpSteer3 (Wang et al., 2025b) Helpfulness 2,000 2,000 04,000 SafeRLHF (Dai et al., 2024) Safety 500 500 01,000 > Total 12,450 12,450 1,461 26,361 • Pairwise: For pairwise evaluation, we adopt the setup of (Guo et al., 2025b) to report the accuracy over a single random ordering of paired responses across all judgment benchmarks. • Listwise: For listwise evaluation in RewardBench2, we follow the best-of-𝑘 setting in (Malik et al., 2025). For example, in best-of-4, the model is provided with a prompt and four candidate completions, and identify the best response among them. Implementation details for baselines. Apart from our backbone models (Qwen-3), we run the following baselines models on our end during evaluation that are publicly available while within our compute budget: • RM-R1 (Chen et al., 2025b): All the models are available at the HuggingFace platform: https: //huggingface.co/collections/gaotang/rm-r1-681128cdab932701cad844c8 . • RRM (Guo et al., 2025b): All the models are available at the HuggingFace platform: https: //huggingface.co/Reward-Reasoning . • Think-RM (Hong et al., 2025): The models at the HuggingFace platform: https://huggingface. co/ilgee/Binary-Think-RM-8B . We chose the binary version due to its reported better performance. • AgentRM (Peng et al., 2025): The codebase of AgentRM is publicly available at https:// github.com/THU-KEG/Agentic-Reward-Modeling . • Gemini-2.5-Flash (Comanici et al., 2025): We follow the guideline at https://ai.google. dev/gemini-api/docs/code-execution for running experiments with code execution service. For RM-R1, RRM, and Think-RM, they are all designed for pairwise ranking only, and we use the same pairwise judging prompt reported in the paper to ensure fair comparison. For other baselines, as some of the works (Whitehouse et al., 2025) are not publicly available, we only use the reported results in the original paper for comparison. E. Detailed Results for Best-of-N Experiments Experiment Setup. Here, we implement three types of the best-of-N selection task. We select AIME-2024, AIME-2025, BigCodeBench and IFeval for evaluation. For AIME-2024 and AIME-2025, each containing 30 problems, we evaluate four backbone models: Gemma-3-27B-It , Qwen-2.5-32B , Qwen-3-32B-Think , and R1-Distill-0528-8B . For each backbone, we allow a maximum gen-eration length of 16k tokens and sample 16 valid responses per problem. For BigCodeBench and > 22 Incentivizing Agentic Reasoning in LLM Judges via Tool-Integrated Reinforcement Learning IFEval, we reuse model outputs from the JETTS dataset (Zhou et al., 2025). On BigCodeBench, we consider Qwen-2.5-32B , DeepSeek-Coder-v2 , and Qwen-2.5-Coder-7B as backbones. For IFEval, we select Qwen-2.5-72B and Qwen-2.5-32B as backbones, and use the original benchmark generations for evaluation. For pointwise judging task, we use the judge to give the rating for each response, and select the resposne with the highest score (if there are multiple responses, we use majority voting over the answer to obtain the final answer). For listwise and pairwise judge task, we follow (Guo et al., 2025b) to adopt a knockout tournament style in ( 𝑂 (𝑛 )) comparisons for promoting efficiency. Detailed Experiment Results. Table 6 reports detailed per-dataset and per-model results, showing the number of solutions passed across four benchmarks under different Best-of-𝑁 judging settings. Table 6 | Performance comparison across benchmarks. Pass@1 and MV@16 are reported alongside different variants of TIR-Judge -Zero (4B/8B, Pointwise/Pairwise/Listwise) and RRM-7B (Pairwise). > Benchmark (Size) Model Pass@1 MV@16 TIR-Judge -Zero 4B Pointwise > TIR-Judge -Zero 4B Pairwise > TIR-Judge -Zero 4B Listwise > TIR-Judge -Zero 8B Pointwise > TIR-Judge -Zero 8B Pairwise > TIR-Judge -Zero 8B Listwise RRM-7B (Pair) AIME 2024 (30) Gemma-3-27B 5910 13 12 11 14 13 11 Qwen-2.5-32B 34498313 12 8Qwen-3-32B-Think 24 26 26 25 24 26 25 24 25 R1-distill-0528-8B 23 24 24 25 24 24 24 24 24 AIME 2025 (30) Gemma-3-27B 68798711 98Qwen-2.5-32B 34612 12 714 10 11 Qwen-3-32B-Think 22 24 25 25 22 25 24 23 24 R1-distill-0528-8B 21 22 22 23 22 22 23 23 23 BigCodeBench (1139) Qwen-3-32B 459 495 569 534 517 550 541 516 515 Deepseek-Coder 285 328 435 427 404 371 448 400 379 Qwen-2.5-7b-Coder 358 401 466 476 447 472 473 446 434 IFEval (541) Qwen-2.5-32B-Instruct 425 436 444 465 458 438 465 450 436 Qwen-2.5-72B-Instruct 446 457 461 478 482 459 476 472 454 From Table 6, we observe that TIR-Judge consistently delivers strong performance across model scales and judging formats, highlighting its robust generalization ability. These results demonstrate that TIR-Judge is not only effective but also readily transferable to diverse target tasks. 23

Google TIR Judge