Official implementation of the ICML 2026 paper: Q-Evolve: Self-evolving LLM Agents with In-distribution Optimization.
Q-Evolve unifies automatic process-reward labeling and policy learning for LLM agents under delayed, episodic rewards. Each iteration trains a critic on hybrid off-policy data, derives step-wise process rewards via advantage estimation, and runs behavior-proximal policy optimization on the same dataset — policy, critic, and dataset co-evolve in a closed loop. Evaluated on ALFWorld, WebShop, and ScienceWorld.
conda create -n qevolve python=3.10 && conda activate qevolve
pip install -r requirements.txt && ./setup.sh
pip install uv && uv pip install "sglang[all]==0.4.8"
pip install vllm==0.8.5 transformers==4.52.3 trl==0.17.0 ray==2.48.0 numpy==1.26.0 \
flashinfer-python==0.2.3 sentencepiece alfworld==0.3.5 scienceworld pyserini \
cleantext openai==0.28.1 gym selenium omegaconf protobuf termcolor colorama \
rank_bm25 matplotlib "tatsu==5.8.3" && pip install --upgrade accelerate
pip install flash-attn --no-build-isolation --no-cache-dir
python -m spacy download en_core_web_lg
cd envs/webshop && python setup.py install && cd ../..Data (expert demos + ALFWorld game files):
huggingface-cli download qlass/qlass_sft_data # expert demonstrations
cd eval_agent/data/alfworld # ALFWorld game files
gdown https://drive.google.com/uc?id=1y7Vqeo0_xm9d3I07vZaP6qbPFtyuJ6kI && unzip alfworld_data.zipPretrained SFT starting checkpoints: huggingface.co/qlass (e.g. qlass-Llama-2-7b-chat-hf-alfworld-sft) — with these you can skip the SFT stage.
One iteration per environment (webshop / alfworld / scienceworld), scripts under bin/llama2-7b/<env>/:
bash iter1/sft.sh # 0) SFT the base agent (or use a released checkpoint)
bash iter1/collect_q_data.sh # 1) explore + build hybrid dataset
bash iter1/token_estimator.sh # 2) train critic, label process rewards (GAE)
bash iter1/policy_extraction.sh # 3) BPPO policy extraction
bash eval.sh # 4) evaluate on the test splitHyperparameters live in bin/llama2-7b/<env>/iter*/config.sh. iter2/ scripts repeat the loop with the evolved policy. Reference hardware: 2–4 GPUs for training, 1 for eval.
Core code: QEvolve/train_critic.py (critic), QEvolve/inf_critic.py (advantage labeling), QEvolve/train_policy.py (BPPO). Agent/environment infrastructure lives in eval_agent/.
@inproceedings{zhang2026qevolve,
title = {Q-Evolve: Self-evolving LLM Agents with In-distribution Optimization},
author = {Zhang, Yudi and Fang, Meng and Chen, Zhenfang and Pechenizkiy, Mykola},
booktitle = {International Conference on Machine Learning (ICML)},
year = {2026}
}