Supported by OSU NLP and NeoCognition, this repo contains open-source code and data reproductions for "Agent Learning via Early Experience".

The paper introduces a training paradigm sitting between imitation learning (IL) and reinforcement learning (RL) — one that is applicable to environments without verifiable reward signals or where long-horizon rollouts make RL impractical. The agent collects its own interaction data by proposing non-expert actions at expert-visited states, then uses the resulting next states as supervision — without any reward.

Two concrete methods under this paradigm:

• Implicit World Modeling (IWM) — train the policy to predict the next state given (state, action) on the rollout data, then continue with imitation on expert data. A two-stage pipeline.
• Self-Reflection (SR) — mix the reflection data (LLM-authored monologues comparing expert vs alternative actions) with expert data, and train with standard next-token prediction.

This repo ships:

• A skill guide (skill/) capturing the general methodology, method mapping, and pitfalls of generating EE data for any env — designed to be consumed by a code agent (any flavor) as a workflow reference.
• Ten env-specific implementation notes (envs/<env>/) that serve two purposes: (1) enough detail to reproduce our numbers on the paper's envs, and (2) worked examples of what "porting a new agentic env into the EE framework" actually looks like. Anyone can pick up the skill and add their own env by following the same recipe.
• Pre-generated SFT data (expert / IWM / reflection) on Hugging Face for every env in this release.

All SFT files (expert / IWM / reflection) are available on huggingface.

The skill/ directory provides a code-agent skill that lets any agentic environment quickly adopt the early-experience learning paradigm — both Implicit World Modeling and Self-Reflection (see the paper for the method itself). More at skill/README.md.

Drop it into your project in one line and point the agent at SKILL.md:

git clone https://github.com/OSU-NLP-Group/EarlyExperience.git && cp -r EarlyExperience/skill /path/to/your/project/

Note that this release uses a unified training setup across all envs — same base model class per env, same optimizer, same effective batch and learning rate — which differs slightly from the per-env hyperparameters described in the paper's Appendix B. All numbers above are the final checkpoint of a single one-shot training run (no ensembling, no re-runs).

Ten envs in this release. Seven with the full loop shipped (data → training → eval); three more in progress.

Text-based household task env (paper §B.1). The agent solves ALFRED-style tasks like "put a clean fork in cabinet" by emitting admissible commands (go to countertop 1, pick up fork 1, ...) in a TextWorld-simulated home. IWM alternatives come from the fully-enumerable admissible-commands list — no LLM proposer needed. See envs/alfworld/README.md for the modification strategy and reproducibility notes.

E-commerce shopping env (paper §B.2). The agent handles instructions like "find a lightweight running jacket under $50" through search[...] and click[...] actions. Alternatives at each expert step are sampled from the state-appropriate action space (alternative product clicks on a results page, alternative query reformulations for search states). See envs/webshop/README.md.

Multi-turn function-calling benchmark (paper §B.3). Because sim state is pure-Python and copy.deepcopy-able, alternative-action probing needs no re-replay or HTTP env spin-up. IWM uses K=10 alt names sampled structurally from the involved sim classes, with one LLM call filling all K argument sets per state. See envs/bfcl_v4/README.md.

Interactive science-lab env (paper §B.6). Requires the admissible-action list at every state; we patch AgentGym's HTTP env-server to expose an /admissible_actions endpoint backed by ScienceWorld's own combination generator. IWM samples K=3 uniform non-expert alts; SR uses policy-proposed alts with refill-from-admissible. See envs/scienceworld/README.md.

Multi-day travel-planning env (paper §B.7). The whole plan is submitted in one turn, which makes SR the dominant methodological challenge: reflections must justify specific slot fills against constraint-stratified alternatives (budget / cuisine / min-nights / city-sequence / mode-chain). See envs/travelplanner/README.md.

Free-form code-action agent env (not in paper). The agent writes Python that runs in a sandboxed REPL with access to mock apps (spotify, venmo, gmail, ...). Expert data comes from the AppWorld SDK's ground-truth programs (no LLM cost), so IWM cost is one proposer call per state plus K env probes. See envs/appworld/README.md.

Text-based Minecraft-style crafting game (not in paper). Admissible actions are entirely client-derivable from (commands_list, inventory) — no server-side patch needed. See envs/textcraft/README.md.

Customer-service env (paper §B.4). Coming soon. See envs/tau-bench/README.md.

Multi-hop question-answering env (paper §B.5). Coming soon. See envs/searchqa/README.md.

Realistic full-stack web-navigation env (paper §B.8). In progress — data released; pipeline description and results coming soon. Expert side is sourced from the Agent Data Protocol (ADP) paper's 4 WebArena-relevant expert sources. See envs/webarena/README.md.

For each env, the linked README describes:

• The upstream env repo (with license and version pin).
• Our high-level modification strategy — the why and what of the changes, not code-level detail. Enough to guide a reimplementation on your own fork.
• The method mapping (how EE's expert / IWM alt / SR reflection concepts realize concretely in this env).
• Data output locations on Hugging Face.
• Key hyperparameters (K, sampling strategy) and re-implementation gotchas.

This repo per se done by Xiangchao Chen, Tinghui Zhu, and Kai Zhang. If you find this code or data helpful, please cite the original paper:

@inproceedings{Zhang2026EarlyExperience,
title={Agent Learning via Early Experience},
author={Kai Zhang and Xiangchao Chen and Bo Liu and Tianci Xue and Zeyi Liao and Zhihan Liu and Xiyao Wang and Yuting Ning and Zhaorun Chen and Xiaohan Fu and Jian Xie and Yuxuan Sun and Boyu Gou and Qi Qi and Zihang Meng and Jianwei Yang and Ning Zhang and Xian Li and Ashish Shah and Dat Huynh and Hengduo Li and Zi Yang and Xuefei Cao and Lawrence Keunho Jang and Shuyan Zhou and Jiacheng Zhu and Huan Sun and Jason E Weston and Yu Su and Yifan Wu},
booktitle={Forty-third International Conference on Machine Learning},
year={2026},
url={https://openreview.net/forum?id=N3dXUHY5dD}
}