DERL
Preprint · 2026

DERL: Dual-Evolving Reinforcement Learning
for Automated Reward Design

Sitao Cheng1, Tianle Li2, Xuhan Huang3, Xunjian Yin4, Difan Zou2

1University of Waterloo  ·  2The University of Hong Kong  ·  3The Chinese University of Hong Kong, Shenzhen  ·  4Duke University

Contact: sitao.cheng@uwaterloo.ca · tianleli@connect.hku.hk · xuhanhuang@link.cuhk.edu.cn

Paper Code Models See the frameworks

TL;DR Automated reward design usually searches with a proposer that never learns: a mutation operator or a frozen LLM. We propose to tune the proposer itself with its trials and their scores. A small proposer composes a reward from task primitives, an inner policy trains with that reward, and the inner policy's validation score updates the proposer by policy gradient.

Read the abstract

Crafting effective reward signals remains a central challenge in reinforcement learning (RL) for complex reasoning. Existing methods for automated reward optimization search with a fixed proposer which never learns from its costly trials. We present Dual-Evolving Reinforcement Learning (DERL), a dual-training framework for autonomous discovery of reward structures. An outer-loop proposer composes a reward from task primitives, and an inner policy trains with this reward. The proposer itself learns by policy gradient from the validation performance of the inner policy. Thus, DERL utilizes past-trial feedback to shape its reward-proposal distribution, an empirical meta-gradient of task success. We validate DERL on embodied (ALFWorld), scientific (ScienceWorld), and mathematical (GSM8K, MATH) reasoning. On multi-turn agent benchmarks, DERL outperforms strong baselines, significantly on out-of-distribution tasks. In mathematics, DERL improves on the outcome reward where directly added primitives hurt. Further analyses under matched compute show that our proposer effectively captures the meta-gradient to discover useful reward signals.

ALFWorld · L2 (OOD)+8.7success rate over RLVMR, the strongest published method
ScienceWorld · L1+8.6success rate over RLVMR, with no extra human labor
Unseen tasks>2×the success rate of the outcome reward at L2 on both agent benchmarks
GSM8K · same compute79.2with a trained 0.5B proposer, against 77.2 for prompted GPT-4o
Reward design frameworks

From hand-written rewards to a proposer that learns

Every framework repeats one costly trial: write a reward, train a policy with it, look at the result. They differ in who writes the next reward and where the result goes.

Animation of three reward design frameworks over seven trials. Traditional: a human revises the reward by hand after each RL run. Static proposer: a frozen LLM proposes rewards and reads earlier scores in its prompt, and the accuracy of its proposals stays flat near 76 to 77 percent. DERL: a trained proposer is updated by policy gradient on each validation score, and the accuracy of its proposals rises from 75.1 to 78.3 percent.
Three ways to design a reward. A human revises the reward after every RL run. A static proposer (a frozen LLM) reads earlier scores in its prompt, but its weights never change. In DERL, every validation score updates the proposer by policy gradient. Curves: mean GSM8K test accuracy of the inner policies trained from each outer step's proposals (Qwen2.5-1.5B inner policy), for the prompted GPT-4o proposer and for DERL-base, from the paper. The reward programs in the animation are illustrative.
Traditional

Human labor in every trial

An expert writes a dense reward or trains a reward model. Each revision needs the expert again, and hand-designed terms invite reward hacking.

Static proposer

Search without learning

Evolution or a prompted LLM writes and refines rewards. The proposer itself never changes, so a trial's outcome acts only through selection or the prompt.

DERL

The proposer evolves too

The proposer is trained by policy gradient on the validation score of the policies its rewards produce. Its weights keep the feedback of all past trials.

Method

Two learners in two loops

An inner loop trains a policy with a proposed reward. An outer loop trains the proposer, with the validation performance of the inner policy as its reward. The two learners co-evolve.

DERL framework. The proposer reads a fixed task instruction and samples n reward programs. Each program trains one inner policy with GRPO, whose validation score is the reward of the program. Group-normalized advantages update the proposer with GRPO.
The DERL framework. The proposer reads a fixed task instruction and samples n reward programs. Each program trains one inner policy with GRPO, and the policy's validation score is the reward of that program. The group-normalized scores update the proposer with GRPO.
Step 1

Compose primitives

The proposer writes an arithmetic expression over executable primitives, such as task success, answer format, or the mean step score over one third of an agent trajectory. A logit mask keeps the output in this grammar.

R = g1 + (g2 · (g3 / 2)) − (g4 / 3)
Step 2

Train an inner policy

Each program is a reward. An inner policy trains with it by GRPO on an inner-training split, and is then scored on a held-out validation split.

vi = Perf(θi; Dval)
Step 3

Update the proposer

The validation scores of one step are group-normalized and update the proposer by policy gradient. Nothing is differentiated through inner training.

Âi = (vi − v̄) / σv

Two variants. DERL-base starts every inner policy from the base model. DERL-full divides the training budget into rounds, and the best policy of each round starts the next one.

Fair evaluation. The search never sees the test set. After the search, a fresh policy trains on the full training set with the found reward, with the same data and no more epochs than the baselines, and is evaluated once.

Results

Better rewards, largest gains on unseen tasks

Embodied and scientific agents

DERL-base beats every baseline at every level on ALFWorld and ScienceWorld, including RLVMR, which adds a hand-designed process reward. The uniform average of the same primitives helps on seen tasks but transfers poorly, while the found rewards more than double the outcome reward on out-of-distribution tasks.

Success rate (%) with Qwen2.5-1.5B-Instruct, mean over four seeds. L0: seen tasks. L1: unseen variants. L2: held-out task types (OOD). GiGPO and RLVMR results are from RLVMR (Zhang et al., 2025).
MethodALFWorldScienceWorld
L0L1L2L0L1L2
GRPO w/ Outcome76.671.129.721.113.710.9
GRPO w/ Average(primitives)88.185.430.537.931.318.0
GiGPO86.783.248.025.815.24.7
RLVMR89.187.956.346.934.426.5
DERL-base91.089.165.047.743.030.1
DERL-full91.888.376.498.295.331.3

Mathematical reasoning

The outcome reward is already a reliable signal here. Adding format and averaged primitives directly raises GSM8K but lowers MATH by up to 4.2%. The rewards found by DERL avoid this trade-off and improve on the outcome reward in both training settings.

Exact match (%) with Qwen2.5-3B, mean over three seeds.
Reward functionGSM8KMATH
Trained on MATH
Outcome82.959.1
Outcome + Format83.956.8
Average(primitives)83.654.9
DERL-base83.260.5
DERL-full84.160.9
Trained on MATH and GSM8K
Outcome82.658.8
Outcome + Format86.455.9
Average(primitives)86.555.8
DERL-base87.060.2
DERL-full87.660.2
Analysis

The gain comes from training the proposer

Same reward space, same compute: only the trained proposer improves

A frozen proposer (random search) and prompted GPT-4o get the same reward space and number of inner runs. Their proposals do not improve over steps. The proposals of the 0.5B DERL proposer rise from 75.1% to 78.3%, and its final policy reaches 79.2%, against 77.2% for GPT-4o and 76.5% for random search.

Mean GSM8K test accuracy of the inner policies (Qwen2.5-1.5B) trained from each outer step's proposals. Hover or focus the chart for every value.
Show the data
Outer stepDERL-baseRLAIFRandom search
075.1376.8876.08
176.6076.6774.90
277.6976.4075.97
377.1376.8554.58
477.2976.4576.34
578.2876.1775.16
678.3076.9574.83

Robust to removed or corrupted primitives

Removing one primitive costs at most 1.3%, and every reduced vocabulary still beats both GRPO baselines. When the format primitive is flipped to penalize correct formatting, the proposer learns a negative coefficient for it and recovers the useful direction.

GSM8K exact match of the final policy (Qwen2.5-1.5B inner policy). Every reduced or flipped vocabulary stays right of the best GRPO baseline.
Show the data
VariantTest exact match (%)
GRPO w/ Outcome75.82
GRPO w/ Average(primitives)76.12
DERL-base, flip g277.79
DERL-base, remove g278.39
DERL-base, remove g378.92
DERL-base, remove g477.94
DERL-base, full vocabulary79.22

Validation and test rise together

Over outer steps, the mean validation score of the inner policies, which is the proposer's reward, rises together with their test score on held-out data. The proposer does not overfit the validation split that supplies its reward.

ALFWorld (L2)

GSM8K

MATH

Each point is the mean over the n inner policies trained from one outer step's proposals, scored on the validation split and on the test set.
Show the data
ALFWorld (L2), success rate (%)
StepVal.Test
051.930.5
172.241.7
264.841.2
372.945.0
473.445.3
575.045.0
670.740.7
783.344.5
886.454.5
954.634.0
1073.443.2
1185.751.8
1285.254.9
1368.243.4
1486.951.7
1586.255.1
GSM8K, accuracy (%)
StepVal.Test
067.859.5
183.073.5
296.686.2
396.686.0
496.486.4
596.886.2
696.886.4
796.586.2
MATH, accuracy (%)
StepVal.Test
048.239.6
159.648.9
268.957.2
368.456.5
467.856.3
568.556.5
669.257.0
770.259.0

The proposer learns stable reward structures

The proposer receives only a scalar score and no structural supervision. Over an ALFWorld L2 run, unstable programs (for example, ones that can penalize success) become rare, and bounded, linear or normalized structures become dominant.

Show the data
Outer stepStable (%)Unstable (%)
062.525.0
175.012.5
250.037.5
375.012.5
462.525.0
575.00.0
687.512.5
787.50.0
887.512.5
Cite

BibTeX

@article{cheng2025derl,
  title   = {Differentiable Evolutionary Reinforcement Learning},
  author  = {Cheng, Sitao and Li, Tianle and Huang, Xuhan and
             Yin, Xunjian and Zou, Difan},
  journal = {arXiv preprint arXiv:2512.13399},
  year    = {2025}
}