🔍 On-Policy Data Evolution for
Visual-Native Multimodal Deep Search Agents

Multimodal deep-search agents need to gather evidence from both text and visual sources, reuse intermediate image observations, and decide when to search, inspect, crop, rotate, or stop. However, static supervised data often fails to cover the evolving tool-use behaviors required by stronger policies.

We present On-Policy Data Evolution (ODE), a training framework for visual-native multimodal deep-search agents. ODE combines a visual-native harness with SFT/RL recipes, reward modules, and on-policy behavior analysis to improve search planning, image-bank reuse, and multimodal evidence grounding.

ODE is built around two components:

• Visual-Native Agent Harness: an interactive multimodal tool-use environment with image-bank memory. Intermediate image observations are stored as reusable references such as <image:0>, enabling later tools to inspect, crop, rotate, or visually search previously discovered visual evidence. The harness unifies web_search, image_search, scholar_search, visit, visual_search, zoom_in, rotation, flip, and python_code in a shared trace format, and the same protocol is used for both evaluation and RL rollouts.
• On-Policy Data Evolution: a training pipeline that combines SFT and RL with policy rollout analysis. SFT teaches the basic action format and tool-use trajectories, while RL optimizes agents over live interactions. Rollout traces are then used to identify behavior gaps and evolve higher-value training mixtures.

The current release includes the training code, evaluation harness, public tool integrations, and scripts needed to reproduce the SFT/RL workflow. The fully automated data-evolution pipeline is reserved in data_evolution/ and marked as coming soon.

ODE improves both 8B and 30B visual-native agents across multimodal deep-search benchmarks.

The visual-native harness enables image observations to be reused by later tools, which is critical for visual evidence grounding.

Static and evolved training mixtures lead to different downstream behavior. ODE uses policy rollouts to select higher-value SFT/RL training signals.

The harness logs fine-grained tool traces, allowing analysis of search, visual inspection, image reuse, and final-answer grounding.

.
├── Megatron-LM/                         # Megatron training dependency
├── mbridge/                             # Megatron bridge dependency
├── ms-swift/                            # SFT training scripts
├── rllm/                                # RL, reward, workflow, and eval harness
│   ├── eval/                            # Visual-native benchmark runner and tools
│   └── vision_deepresearch_async_workflow/
│       ├── data_prepare/                # RL data conversion and registration
│       └── run/                         # RL training scripts
├── verl/                                # vendored verl dependency
├── vision_deepresearch_async_workflow/  # top-level workflow compatibility package
├── data_evolution/                      # coming soon
└── figs/                                # paper and README assets

git clone <this-repo-url>
cd <repo>

# Install verl.
cd rllm/verl
pip install -e .

# Install Megatron-LM.
cd ../../Megatron-LM
pip install -e .

# Install mbridge.
cd ../mbridge
pip install -e .

# Install rLLM.
cd ../rllm
pip install -e .

# Install public tool dependencies.
pip install requests==2.32.3 certifi pillow

# Return to project root.
cd ..

For SFT training, install ms-swift:

cd ms-swift
pip install -e .
cd ..

The open-source harness uses public providers:

• web_search, image_search, scholar_search, and visual_search use Google results through Serper.
• visit uses Jina Reader.
• zoom_in, rotation, flip, and python_code run locally.

visual_search requires a public image URL. This release uploads temporary images to a user-owned public GitHub repository through the GitHub Contents API, then sends the raw image URL to Serper Lens.

export SERPER_API_KEY=...
export JINA_API_KEY=...                         # optional
export VISUAL_SEARCH_GITHUB_TOKEN=...
export VISUAL_SEARCH_GITHUB_REPO=<GitHubUser>/<PublicRepo>
export VISUAL_SEARCH_GITHUB_BRANCH=main
export VISUAL_SEARCH_GITHUB_UPLOAD_DIR=img

Do not commit API keys or GitHub tokens.

RL training uses RewardDeepResearchFn to score final answers. Exact matches are handled locally; semantic matches can be judged by an OpenAI-compatible LLM endpoint.

For an OpenAI-compatible judge:

export JUDGE_API_KEY=...
export JUDGE_BASE_URL=https://api.openai.com/v1
export JUDGE_MODEL=gpt-5.2

For a local vLLM judge:

export JUDGE_URL=http://127.0.0.1:8000/v1
export JUDGE_MODEL=Qwen/Qwen3-VL-30B-A3B-Instruct

The judge also accepts OPENAI_API_KEY or TOGETHER_AI_API_KEY as fallbacks when JUDGE_API_KEY is not set. Optional tuning variables include JUDGE_TEMPERATURE, JUDGE_MAX_TOKENS, JUDGE_TIMEOUT, JUDGE_RETRY_ATTEMPTS, and JUDGE_FALLBACK_MODELS.

SFT data should be converted to the JSONL format expected by ms-swift. If your source data is parquet with image bytes, use the conversion helper:

bash ms-swift/run/data_prepare/convert_parquet2jsonl.sh

The resulting records should point to local image files and contain the multimodal conversations required by the selected Qwen-VL template.

RL data should contain question, answer, and images fields. Convert parquet data if needed:

bash rllm/vision_deepresearch_async_workflow/data_prepare/convert_parquet2jsonl.sh

Then register the dataset:

cd rllm/vision_deepresearch_async_workflow/data_prepare
python register_rl_dataset.py \
  --jsonl_path data/rl/train.jsonl \
  --register_name Vision-DeepResearch-QA \
  --train_ratio 0.9

data_evolution/ is reserved for the automated on-policy data-evolution pipeline.

Status: coming soon.

8B:

MODEL_PATH=Qwen/Qwen3-VL-8B-Instruct \
DATASET_PATH=data/sft/train.jsonl \
OUTPUT_DIR=outputs/sft_8b \
bash ms-swift/run/vision_deepresearch_SFT_8B_megatron_lr2e5_2ep.sh

30B-A3B:

MODEL_PATH=Qwen/Qwen3-VL-30B-A3B-Instruct \
DATASET_PATH=data/sft/train.jsonl \
OUTPUT_DIR=outputs/sft_30b_a3b \
bash ms-swift/run/vision_deepresearch_SFT_30B_A3B_megatron_lr2e5_2ep.sh

Set public tool environment variables before launching RL:

export SERPER_API_KEY=...
export JINA_API_KEY=...
export VISUAL_SEARCH_GITHUB_TOKEN=...
export VISUAL_SEARCH_GITHUB_REPO=<GitHubUser>/<PublicRepo>
export VISUAL_SEARCH_GITHUB_BRANCH=main
export VISUAL_SEARCH_GITHUB_UPLOAD_DIR=img
export JUDGE_API_KEY=...
export JUDGE_BASE_URL=https://api.openai.com/v1
export JUDGE_MODEL=gpt-5.2

8B:

MODEL_PATH=Qwen/Qwen3-VL-8B-Instruct \
DATASET_NAME=Vision-DeepResearch-QA \
CKPTS_DIR=outputs/rl_8b \
bash rllm/vision_deepresearch_async_workflow/run/train_8b_grpo.sh

30B-A3B:

MODEL_PATH=Qwen/Qwen3-VL-30B-A3B-Instruct \
DATASET_NAME=Vision-DeepResearch-QA-30B \
CKPTS_DIR=outputs/rl_30b_a3b \
bash rllm/vision_deepresearch_async_workflow/run/train_30b_a3b_grpo.sh

Start an OpenAI-compatible model server, for example with vLLM:

CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 vllm serve \
  Qwen/Qwen3-VL-8B-Instruct \
  --host 0.0.0.0 \
  --port 8002 \
  --tensor-parallel-size 8 \
  --gpu-memory-utilization 0.8 \
  --served-model-name Qwen3-VL-8B-Instruct \
  --max_model_len 160000 \
  --mm-processor-cache-gb 0 \
  --no-enable-prefix-caching

Run evaluation:

cd rllm
export SERPER_API_KEY=...
export JINA_API_KEY=...
export VISUAL_SEARCH_GITHUB_TOKEN=...
export VISUAL_SEARCH_GITHUB_REPO=<GitHubUser>/<PublicRepo>

python3 -m eval.eval_runner \
  --parquet data/bench/example.parquet \
  --base-url http://127.0.0.1:8002/v1 \
  --model Qwen3-VL-8B-Instruct \
  --api-key EMPTY \
  --parallel-tasks 8 \
  --run-name example_eval

The benchmark parquet format is documented in rllm/eval/README.md.

This implementation builds on several excellent open-source projects:

• Vision-DeepResearch
• verl
• ms-swift
• rLLM

If you find this work helpful, please cite our paper:

@article{huang2026onpolicydataevolution,
  title={Towards On-Policy Data Evolution for Visual-Native Multimodal Deep Search Agents},
  author={Huang, Shijue and Guo, Hangyu and Li, Chenxin and Lu, Junting and Geng, Xinyu and Su, Zhaochen and Li, Zhenyu and Chen, Shuang and Wang, Hongru and Fung, Yi R.},
  journal={arXiv preprint arXiv:2605.10832},
  year={2026}
}