[Roadmap] 2026 H2 Milestones

[sitabulaixizawaluduo]

AReaL 2026 H2 Milestone Tracker

Introduction

This document tracks major planned enhancements for AReaL through the second half of 2026. Our development roadmap is organized into two categories to help contributors identify where they can make the most impact:

• On-going: Features currently under active development by AReaL maintainers. These are our immediate priorities.
• Planned but not in progress: Features that are good to have but currently lacking bandwidth.

Tips for new contributors: We use [CC] as a type annotation to mark items suitable for community contributions. If you're interested in contributing, please reach out to discuss implementation details.

---

Backends

On-going

• Support training the latest large MoE models with the Megatron backend, (including Kimi 2.5 / GLM 5 / DeepSeek V3 / Qwen 3.6, etc.) feat(mcore): add Bailing-MoE V2.5 megatron-bridge adapter #1372 feat(mcore): add GLM-5/DeepSeek-V3 model support (mbridge + megatron-bridge) #1373
• More distribution strategies for colocated deployment

Planned but not in progress

• Omini model RL support with FSDP backend
• Proxy server support via RayScheduler (HTTP launcher actor) [CC]
• Weight delta update with awex
• Memory service for self-evolving agent
• Support Openenv in Agentic RL

---

Usability

On-going

• Effective AReaL CLI support [Feature] Operator CLI (areal) for the v2 microservice architecture #1374
• Profiling toolkit (distributed trace)
• Online RL training example based on AReaL 2.0 architecture

Planned but not in progress

• More RL post-training paradigms: IcePop Plus algorithm (verified in Ling MoE training) and stepwise reward rubrics
• GUI Agent VLM training example
• OS-bench VLM training example [CC]
• Diffusion image/video generation model RL post-training [CC]
• Comprehensive LoRA support over the existing policy model training.
• AReaL autopilot (automatic performance-optimized RL deployment suggestions and systematic bottleneck analysis)

---

Documentation

Planned but not in progress

• Explain benchmarking results from nightly CI and how to extract issues

[HT-Yuan]

@sitabulaixizawaluduo

Hello,I am currently working on implementing a full-duplex Agent built upon the Omni model for my own project, and my development direction coincides with several planned enhancements of AReaL, including:
1）Omni model RL support with FSDP backend
2）Memory service for self-evolving agent
I would like to clarify the specific definition of Memory service for self-evolving agent in this context.
In conventional Agent implementations, memory is generally implemented as an auxiliary plug-in module, with its content compressed or updated relying on an LLM. For this reason, I have not yet figured out how this memory component can be integrated with RL.

Looking forward to your reply, thank you.

[sitabulaixizawaluduo]

@sitabulaixizawaluduo

Hello,I am currently working on implementing a full-duplex Agent built upon the Omni model for my own project, and my development direction coincides with several planned enhancements of AReaL, including: 1）Omni model RL support with FSDP backend 2）Memory service for self-evolving agent I would like to clarify the specific definition of Memory service for self-evolving agent in this context. In conventional Agent implementations, memory is generally implemented as an auxiliary plug-in module, with its content compressed or updated relying on an LLM. For this reason, I have not yet figured out how this memory component can be integrated with RL.

Looking forward to your reply, thank you.

Thank you for your attention. The biggest change in AReaL v2 is that it has adopted a micro service architecture. Based on this structure, users can not only perform RL training but also build agent services. The memory service is part of this component. Similarly, we also plan to integrate the memory service into Online RL training.

[Tiagosf00]

您好 @sitabulaixizawaluduo, I’m curious about the tracing part “Profiling toolkit (distributed trace)”. Is the current direction to extend AReaL’s existing perf_tracer to the new microservice architecture, or to integrate with an external tracing framework like OpenTelemetry/Grafana/Jaeger?

[sitabulaixizawaluduo]

您好 @sitabulaixizawaluduo, I’m curious about the tracing part “Profiling toolkit (distributed trace)”. Is the current direction to extend AReaL’s existing perf_tracer to the new microservice architecture, or to integrate with an external tracing framework like OpenTelemetry/Grafana/Jaeger?

Thank you for your attention, referring to the latter, there will be an external framework to handle this matter.

[Fyrgo8]

Hi @sitabulaixizawaluduo,

I'd like to work on "Diffusion image/video generation model RL post-training". I recently submitted PR #1407 (trajectory dump/replay, currently under review) and am looking to take this on as my next contribution.

My rough plan:

Phase 1 (initial PR): Image generation only. Minimal PoC with SD 1.5 + LoRA + GRPO + aesthetic reward. GRPO is chosen first because it doesn't require a value function (lightest implementation path) and reuses the existing GRPOConfig infrastructure. The PR would include:

• A DiffusionRolloutWorkflow that records per-step log_probs via DDIM-with-eta sampling
• A DiffusionInferenceEngine wrapping diffusers
• An end-to-end example script

All code under areal/experimental/diffusion/, zero changes to existing LLM paths.

Phase 2: FSDP2 sharding for UNet/DiT (with architecture-aware sharding strategies for skip connections vs. regular transformer blocks), flow matching model support (SD3.5, ODE-to-SDE), multiple reward functions (CLIP score for text-image alignment, safety classifier, HPSv2), and algorithm variants (DDPO/PPO-clip alongside GRPO). Video generation (CogVideoX / HunyuanVideo) as a future extension once the image pipeline is stable.

For the API, my inclination is to create separate DiffusionModelRequest/DiffusionModelResponse classes and a DiffusionInferenceEngine base — the current token-centric interfaces don't map naturally to denoising trajectories, and extending them with optional fields would pollute the LLM path. Happy to adjust if you have a different preference.

The main references I'm building on are Flow-GRPO (NeurIPS 2025) and DanceGRPO for the algorithm, and ddpo-pytorch / TRL DDPOTrainer for the log_prob computation pattern.

One question: Would a single-GPU-verified Phase 1 PoC be acceptable as the initial PR, or do you expect distributed support from the start?

Happy to open a draft PR early for feedback. Thanks!

[sitabulaixizawaluduo]

Hi @sitabulaixizawaluduo,

I'd like to work on "Diffusion image/video generation model RL post-training". I recently submitted PR #1407 (trajectory dump/replay, currently under review) and am looking to take this on as my next contribution.

My rough plan:

Phase 1 (initial PR): Image generation only. Minimal PoC with SD 1.5 + LoRA + GRPO + aesthetic reward. GRPO is chosen first because it doesn't require a value function (lightest implementation path) and reuses the existing GRPOConfig infrastructure. The PR would include:

• A DiffusionRolloutWorkflow that records per-step log_probs via DDIM-with-eta sampling
• A DiffusionInferenceEngine wrapping diffusers
• An end-to-end example script

All code under areal/experimental/diffusion/, zero changes to existing LLM paths.

Phase 2: FSDP2 sharding for UNet/DiT (with architecture-aware sharding strategies for skip connections vs. regular transformer blocks), flow matching model support (SD3.5, ODE-to-SDE), multiple reward functions (CLIP score for text-image alignment, safety classifier, HPSv2), and algorithm variants (DDPO/PPO-clip alongside GRPO). Video generation (CogVideoX / HunyuanVideo) as a future extension once the image pipeline is stable.

For the API, my inclination is to create separate DiffusionModelRequest/DiffusionModelResponse classes and a DiffusionInferenceEngine base — the current token-centric interfaces don't map naturally to denoising trajectories, and extending them with optional fields would pollute the LLM path. Happy to adjust if you have a different preference.

The main references I'm building on are Flow-GRPO (NeurIPS 2025) and DanceGRPO for the algorithm, and ddpo-pytorch / TRL DDPOTrainer for the log_prob computation pattern.

One question: Would a single-GPU-verified Phase 1 PoC be acceptable as the initial PR, or do you expect distributed support from the start?

Happy to open a draft PR early for feedback. Thanks!

Thank you for your contribution! I have seen the related PR and will provide feedback as soon as possible. To answer your question, I think it's okay to use a single GPU as the initial version, which is a step towards achieving efficient performance.

[HughLLiu]

Hi, @sitabulaixizawaluduo , I’m interested in working on this [CC] item if it is still available.

I’m new to contributing to this community, so I’d like to first check whether my
understanding matches the intended design.

From reading the current implementation, the proxy server logic already exists in
areal.experimental.openai.proxy.proxy_rollout_server. LocalScheduler and
SlurmScheduler support this by honoring the command argument in
fork_workers(): when RolloutController.start_proxy() passes the proxy server
module, they fork a colocated child process that directly runs
python -m areal.experimental.openai.proxy.proxy_rollout_server ....

For RayScheduler, however, fork_workers(..., command=...) currently ignores
command and always creates RayRPCServer actors. So my understanding is that
the missing piece is not the proxy logic itself, but a Ray-side HTTP launcher
actor that can start and manage the existing proxy_rollout_server module on
the appropriate Ray node / placement group.

A possible implementation direction would be to add a lightweight Ray-managed
HTTP launcher actor, analogous to the Local/Slurm guard /fork path. It would:

• be scheduled next to the target rollout worker;
• launch the existing proxy_rollout_server module as a subprocess;
• wait for /health;
• return the proxy server host/port to RayScheduler;
• clean up the subprocess when the worker is deleted.

Does this match the maintainers’ intended direction for this item? If yes, I’d
be happy to work on an initial PR.