When Vision Overrides Language: Evaluating and Mitigating Counterfactual Failures in VLAs

Video Summary

Overview

(a) Vision-Language-Action Models (VLAs) often suffer from counterfactual failures due to vision shortcuts, defaulting to well-learned scene-specific behaviors instead of faithfully following instructions. (b) We study this issue and introduce LIBERO-CF, the first counterfactual benchmark for evaluating language following in VLAs. (c) We propose Counterfactual Action Guidance (CAG), a dual-branch inference scheme that mitigates counterfactual failures in VLAs. (d) Extensive experiments in both simulation and real-world experiments demonstrate the effectiveness of CAG across diverse VLAs.

Abstract

Vision-Language-Action models (VLAs) promise to ground language instructions in robot control, yet in practice often fail to follow language faithfully. When presented with instructions that lack strong scene-specific supervision, VLAs suffer from counterfactual failures: they act based on vision shortcuts induced by dataset biases, repeatedly executing well-learned behaviors and selecting objects frequently seen during training regardless of language intent.

To support systematic study, we introduce LIBERO-CF, the first counterfactual benchmark for VLAs that evaluates language following capability by assigning alternative, feasible task instructions under the same LIBERO scene layouts. Our evaluation reveals that counterfactual failures are prevalent yet underexplored across state-of-the-art VLAs.

To mitigate these failures, we propose >Counterfactual Action Guidance (CAG), a simple yet effective dual-branch inference scheme that explicitly strengthens language conditioning in VLAs. CAG combines a standard VLA policy with a parallel language-unconditioned branch, amplifying the difference between their action predictions induced by language instructions. This design reduces reliance on vision shortcuts, improves robustness on counterfactual tasks, and requires neither additional demonstrations nor modifications to the base VLA architecture or its pretrained weights.

Extensive experiments demonstrate consistent improvements across diverse VLAs. On LIBERO-CF, our training-free variant, TF-CAG, improves π_0.5 by 9.7% in language grounding and 3.6% in task success. When using a separate Vision-Action (VA) module, VA-CAG achieves larger gains of 15.5% and 8.5%, respectively. In real-world evaluations, VA-CAG improves language grounding by 9.4% and task success by 17.2% on average.

Evidence for Vision Shortcuts in VLAs

Input Modality Ablation

Success Rate (%)

Model	Input	Spatial	Object	Goal	Long	Avg.
OpenVLA-OFT	Vision + Language	97.6	98.4	97.9	94.5	97.1
OpenVLA-OFT	Vision-only	83.2	98.2	10.0	85.0	69.1
π₀	Vision + Language	96.8	98.8	95.8	85.2	94.2
π₀	Vision-only	28.0	26.8	3.4	15.6	18.5
π_0.5	Vision + Language	98.8	98.2	98.0	92.4	96.9
π_0.5	Vision-only	68.2	56.6	10.2	71.6	51.7

In input modality ablation, all evaluated VLAs retain high performance after removing language, indicating that vision alone can induce strong task-specific action preferences.

Distribution of Grasp Positions

We visualize the distribution of grasp positions from 50 trials as heatmaps under different instructions. Even when given counterfactual or empty instructions, VLAs tend to execute the well-learned training task in the scene.

Distraction by Training-Task Objects

Removing the training-task object in the scene improves the success rates of VLAs on counterfactual instructions.

Method

We propose counterfactual action guidance (CAG), a simple yet effective inference-time action mixing scheme for enhancing language conditioning in VLAs. With a standard VLA branch and a language-unconditioned branch, CAG estimates and amplifies the language-induced action shift beyond vision-only preferences.

To ensure broad applicability across different VLA architectures, CAG operates directly on the final predicted action chunks after both branches complete their inference. This provides a unified inference-time action-space guidance mechanism for both flow-based and deterministic regression policies, without modifying the original VLA architectures or pretrained weights.

We consider two strategies:

Training-Free Strategy (TF): Directly use the VLA with no language input as the language-unconditioned branch.
Training a Vision-Action Prior (VA): Train a separate Vision-Action (VA) Model to serve as the language-unconditioned branch.

Simulation Experiments on the LIBERO-CF Benchmark

Success Rate (%)

Model	Metric	CAG	CF-Spatial		CF-Object		CF-Long		CF-OOD		Average
Model	Metric	CAG	Faithful ↑	Biased ↓	Faithful ↑	Biased ↓	Faithful ↑	Biased ↓	Faithful ↑	Biased ↓	Faithful ↑	Biased ↓
OpenVLA-OFT	Grounding	/	6.8	84.7	2.0	99.6	4.0	62.9	6.0	87.3	4.7	83.6
		TF	11.1	78.5	1.6	100.0	4.9	61.1	5.6	88.1	5.8	81.9
		VA	26.5	59.5	3.8	91.8	6.9	60.0	7.9	50.1	11.3	65.4
	Success	/	1.1	80.7	0.0	97.6	0.2	54.4	0.1	81.7	0.4	78.6
		TF	3.3	73.3	0.0	94.4	0.2	54.0	0.1	83.5	0.9	76.3
		VA	7.9	35.2	0.0	57.0	0.4	48.9	0.1	22.7	2.1	41.0
π₀	Grounding	/	70.9	37.1	6.0	59.4	20.4	83.1	18.0	72.4	28.8	63.0
		TF	74.5	32.5	12.2	50.6	19.6	81.8	21.1	72.4	31.9	59.3
		VA	85.2	24.4	13.8	53.8	22.7	82.2	27.6	47.5	37.3	52.0
	Success	/	34.4	19.1	0.0	45.0	1.1	63.1	2.7	52.9	9.6	45.0
		TF	33.5	14.4	0.6	15.0	2.0	38.7	2.1	42.3	9.6	27.6
		VA	36.7	6.1	0.0	29.2	2.9	48.7	2.1	19.2	10.4	25.8
π_0.5	Grounding	/	39.3	61.3	11.4	68.8	51.6	58.4	20.7	74.0	30.8	65.6
		TF	52.0	45.3	24.6	65.2	60.7	56.7	24.8	71.9	40.5	59.8
		VA	50.7	45.2	34.0	49.8	64.2	61.1	36.4	52.8	46.3	52.2
	Success	/	24.4	56.9	5.8	66.8	15.8	50.4	6.9	69.6	13.2	60.9
		TF	27.3	33.3	9.2	41.4	20.9	36.7	9.9	62.7	16.8	43.5
		VA	31.6	36.0	18.0	29.8	26.7	41.8	10.3	37.1	21.7	36.2

Faithful: when the model follows the instructed task, Biased: when the model follows the original training task.

π_0.5

Ours

Quantitative Results

Citation

@article{fang2026when,
  title={When Vision Overrides Language: Evaluating and Mitigating Counterfactual Failures in VLAs},
  author={Fang, Yu and Feng, Yuchun and Jing, Dong and Liu, Jiaqi and Yang, Yue and Wei, Zhenyu and Szafir, Daniel and Ding, Mingyu},
  journal={arXiv preprint arXiv:2602.17659},
  year={2026}
}