When Vision Overrides Language: Evaluating and Mitigating Counterfactual Failures in VLAs

CoRL 2026 (Spotlight)

Yu Fang, Yuchun Feng, Dong Jing, Jiaqi Liu, Yue Yang, Zhenyu Wei, Daniel Szafir, Mingyu Ding
University of North Carolina at Chapel Hill

This repository contains the two parts of our paper:

• LIBERO-CF: a benchmark that tests whether Vision-Language-Action (VLA) models follow counterfactual instructions.
• Counterfactual Action Guidance (CAG): an inference-time method that mitigates counterfactual failures in VLAs.

• Part 1: LIBERO-CF
• Part 2: Counterfactual Action Guidance (CAG)
• Repository Structure
• Citation
• Acknowledgement
• License

LIBERO-CF is a counterfactual benchmark built on top of LIBERO for testing whether Vision-Language-Action (VLA) models can correctly follow counterfactual instructions. By evaluating a set of plausible conditions for each task, LIBERO-CF probes whether a policy truly grounds language in the scene rather than relying on memorized behaviors.

The conditions for each task are defined in libero/libero/conditions/. The evaluation logs two metrics for every condition:

• Touch rate: whether the gripper touches the object associated with the condition. We use this contact to indicate whether the policy follows the instruction for that condition
• Success rate: whether the condition is successfully completed

The paper reports results for two of these conditions per task: the instructed (counterfactual) task and the training task. The grounding rate is the touch rate of the corresponding condition.

LIBERO-CF does not require additional data collection beyond the standard LIBERO dataset (LIBERO-Spatial, LIBERO-Object, LIBERO-Goal, LIBERO-10). All task suites are designed to evaluate models that have been fine-tuned on the standard LIBERO dataset. The benchmark probes whether those fine-tuned policies generalize beyond their training scenarios.

Clone the repository:

git clone https://github.com/yuffish/VLA-CF.git
cd VLA-CF

Create and activate a virtual environment with uv:

uv venv --python 3.8 --prompt libero_cf .venv
source .venv/bin/activate

Install dependencies:

uv pip sync requirements.txt requirements_libero.txt \
    --extra-index-url https://download.pytorch.org/whl/cu113 \
    --index-strategy=unsafe-best-match
uv pip install -e packages/openpi-client
uv pip install -e .

The policy runs in its own environment, and the benchmark connects to it over a WebSocket.

π0 / π0.5 (openpi)

Set up the openpi environment from openpi-cf/:

cd openpi-cf
GIT_LFS_SKIP_SMUDGE=1 uv sync
GIT_LFS_SKIP_SMUDGE=1 uv pip install -e .

Serve the policy:

# π0.5 (default for --env LIBERO)
CUDA_VISIBLE_DEVICES=<ID> XLA_PYTHON_CLIENT_PREALLOCATE=false uv run scripts/serve_policy.py \
    --port <port> \
    --env LIBERO


# π0
CUDA_VISIBLE_DEVICES=<ID> XLA_PYTHON_CLIENT_PREALLOCATE=false uv run scripts/serve_policy.py \
    --port <port> \
    policy:checkpoint \
        --policy.config=pi0_libero \
        --policy.dir=gs://openpi-assets/checkpoints/pi0_libero

OpenVLA-OFT

Set up the openvla-oft environment by following the OpenVLA-OFT setup instructions. Then copy openvla-oft-cf/ over the root of your OpenVLA-OFT repository and install the remaining dependencies:

conda activate openvla-oft
cd /path/to/openvla-oft
cp -r /path/to/VLA-CF/openvla-oft-cf/. .
pip install -e packages/openpi-client
pip install tyro

Serve the policy:

CUDA_VISIBLE_DEVICES=<ID> python serving/serve_openvla_oft.py \
    --model_path moojink/openvla-7b-oft-finetuned-libero-spatial-object-goal-10 \
    --task_suite_name <libero_suite> \
    --port <port>

<libero_suite> selects the checkpoint's normalization statistics and must be the standard LIBERO suite that the CF suite is based on, such as libero_spatial for libero_cf_spatial.

In a separate terminal, from the root of this repository, run the evaluation (use --args.replan_steps 8 for OpenVLA-OFT):

source .venv/bin/activate
export PYTHONPATH=$PYTHONPATH:$PWD
export LIBERO_CONFIG_PATH=$PWD/.libero

# Replace <suite> with one of: libero_cf_spatial, libero_cf_object, libero_cf_long,
#   libero_cf_spatial_focused, libero_cf_ood
python eval/main_cf.py \
    --args.task_suite_name <suite> \
    --args.port <port>

Rollouts are saved to rollouts/<suite>-<timestamp>/.

To visualize rollouts, run:

python eval/batch_visualize.py --rollouts_dir rollouts/<your_rollouts_dir>

This generates per-task videos, end-effector trajectories, and gripper-close heatmaps.

All examples below use 50 rollouts per task on an NVIDIA RTX 6000 (~17 min per task).

Task: 01-pick_up_the_cookie_box_and_place_it_on_the_plate

Condition 'on cookies_1 plate_1'          touch rate: 35/50 (70.0%)   success rate: 25/50 (50.0%)
Condition 'on akita_black_bowl_1 plate_1' touch rate: 16/50 (32.0%)   success rate: 15/50 (30.0%)

Task: 01-pick_up_the_cookie_box_and_place_it_on_the_plate

Condition 'on cookies_1 plate_1'           touch rate: 49/50 (98.0%)   success rate: 33/50 (66.0%)
Condition 'on akita_black_bowl_2 plate_1'  touch rate:  1/50  (2.0%)   success rate:  0/50  (0.0%)

Task: 01-put_the_wine_bottle_in_the_bottom_drawer_of_the_cabinet

Condition 'on wine_bottle_1 white_cabinet_1_bottom_region'        touch rate: 34/50 (68.0%)   success rate: 2/50 (4.0%)
Condition 'on akita_black_bowl_1 white_cabinet_1_bottom_region'   touch rate: 21/50 (42.0%)   success rate: 0/50 (0.0%)

Grounding and success rates (%) on LIBERO-CF. F. (Faithful) = rate on the instructed condition; B. (Biased) = rate on the training-task condition. Higher F. and lower B. are better.

Counterfactual Action Guidance is a dual-branch, inference-time scheme that strengthens language conditioning in VLAs. CAG combines a standard VLA policy with a language-unconditioned module, amplifying the language-induced difference between their action predictions. We provide example implementations for openpi (openpi-cf/) and OpenVLA-OFT (openvla-oft-cf/), in two versions:

• TF (training-free): the unconditioned branch is the VLA itself with its language input removed, so no additional model is needed.
• VA (Vision-Action): the unconditioned branch is a separately trained Vision-Action model, available on Hugging Face.

Set up the environments as described in Serve a policy.

Serve the CAG policy in the openpi environment.

TF version (training-free)

# π0
CUDA_VISIBLE_DEVICES=<ID> XLA_PYTHON_CLIENT_PREALLOCATE=false uv run scripts/serve_policy_TF.py \
    --port <port> \
    --guidance_scale 2.0 \
    policy:checkpoint \
        --policy.config=pi0_libero \
        --policy.dir=gs://openpi-assets/checkpoints/pi0_libero


# π0.5
CUDA_VISIBLE_DEVICES=<ID> XLA_PYTHON_CLIENT_PREALLOCATE=false uv run scripts/serve_policy_TF.py \
    --port <port> \
    --guidance_scale 2.0 \
    policy:checkpoint \
        --policy.config=pi05_libero \
        --policy.dir=gs://openpi-assets/checkpoints/pi05_libero

VA version (with Vision-Action model)

Download the VA model for π0 from VLA-CF/pi0-libero-VA and for π0.5 from VLA-CF/pi05-libero-VA.

# π0
CUDA_VISIBLE_DEVICES=<ID> XLA_PYTHON_CLIENT_PREALLOCATE=false uv run scripts/serve_policy_VA.py \
    --guidance_scale 1.5 \
    --port <port> \
    policy:checkpoint \
        --policy.config=pi0_libero \
        --policy.dir=gs://openpi-assets/checkpoints/pi0_libero \
    uncond-policy:checkpoint \
        --uncond_policy.config=pi0_libero_unconditioned \
        --uncond_policy.dir=/path/to/pi0-libero-VA


# π0.5
CUDA_VISIBLE_DEVICES=<ID> XLA_PYTHON_CLIENT_PREALLOCATE=false uv run scripts/serve_policy_VA.py \
    --guidance_scale 1.5 \
    --port <port> \
    policy:checkpoint \
        --policy.config=pi05_libero \
        --policy.dir=gs://openpi-assets/checkpoints/pi05_libero \
    uncond-policy:checkpoint \
        --uncond_policy.config=pi05_libero_unconditioned \
        --uncond_policy.dir=/path/to/pi05-libero-VA

Serve the CAG policy in the openvla-oft environment. <libero_suite> is a standard LIBERO suite, as described in Serve a policy.

TF version (training-free)

CUDA_VISIBLE_DEVICES=<ID> python serving/serve_openvla_oft_TF.py \
    --model_path moojink/openvla-7b-oft-finetuned-libero-spatial-object-goal-10 \
    --task_suite_name <libero_suite> \
    --guidance_scale 2.0 \
    --port <port>

VA version (with Vision-Action model)

Download the VA model from VLA-CF/openvla-oft-libero-VA.

CUDA_VISIBLE_DEVICES=<ID> python serving/serve_openvla_oft_VA.py \
    --conditioned_model_path moojink/openvla-7b-oft-finetuned-libero-spatial-object-goal-10 \
    --unconditioned_model_path /path/to/openvla-oft-libero-VA \
    --task_suite_name <libero_suite> \
    --guidance_scale 2.0 \
    --port <port>

In a separate terminal, from the root of this repository, run the evaluation against the CAG server:

source .venv/bin/activate
export PYTHONPATH=$PYTHONPATH:$PWD
export LIBERO_CONFIG_PATH=$PWD/.libero

# Replace <suite> with one of: libero_cf_spatial, libero_cf_object, libero_cf_long,
#   libero_cf_spatial_focused, libero_cf_ood

# π0 / π0.5
python eval/main_cf.py \
    --args.task_suite_name <suite> \
    --args.port <port>


# OpenVLA-OFT
python eval/main_cf.py \
    --args.task_suite_name <suite> \
    --args.port <port> \
    --args.replan_steps 8

VLA-CF/
├── eval/
│   ├── main_cf.py            # Main evaluation script
│   ├── batch_visualize.py    # Batch visualization of saved rollouts
│   └── visualize.py          # Per-rollout trajectory and heatmap rendering
├── libero/
│   └── libero/
│       ├── bddl_files/       # Task definitions (.bddl) for each suite
│       ├── conditions/       # Per-task counterfactual condition specs (.json)
│       └── init_files/       # Initial simulator states per suite
├── openpi-cf/                # openpi with CAG policy serving (π0 / π0.5)
├── openvla-oft-cf/           # OpenVLA-OFT files with CAG policy serving, copied over the upstream repository
├── packages/
│   └── openpi-client/        # WebSocket client for openpi-compatible servers
└── scripts/                  # Dataset and task creation utilities

If you find our work useful, please cite:

@article{fang2026when,
  title={When Vision Overrides Language: Evaluating and Mitigating Counterfactual Failures in VLAs},
  author={Fang, Yu and Feng, Yuchun and Jing, Dong and Liu, Jiaqi and Yang, Yue and Wei, Zhenyu and Szafir, Daniel and Ding, Mingyu},
  journal={arXiv preprint arXiv:2602.17659},
  year={2026}
}

This work builds upon excellent open-source projects:

• LIBERO
• openpi
• OpenVLA-OFT

We gratefully acknowledge the authors of these projects.

This project is released under the MIT License. Task assets follow the licensing terms of the original LIBERO dataset.