Embodied planning increasingly relies on vision-language models (VLMs) to translate instructions and visual observations into executable action sequences. However, fluent plans are not always executable. A VLM may refer to objects that are not visually observed, select actions whose required affordances are unavailable, or violate syntax and action constraints. We introduce CLAMP, a multimodal constraint-grounding framework that turns scene evidence into decoding-time constraints for a frozen VLM planner. CLAMP uses the initial observation to restrict object references to those supported by the scene, while a provided symbolic action model specifies state transitions and goals. During decoding, hard masks eliminate invalid next-token candidates, while a Hidden Markov Model (HMM)-based world-state lookahead module reweights the probabilities of the remaining feasible candidates based on action preconditions and goal reachability. This allows the planner to retain the VLM’s language prior while preventing visually unsupported, unsafe, or infeasible candidates from entering the plan. For unseen tasks and environments, CLAMP adapts the HMM at test time using label-free continuations sampled from the frozen VLM. Experiments on VLABench, SafeAgentBench, and TaPA show that scene-grounded constraints improve object grounding and safety, while most remaining failures stem from perception errors or misaligned constraint specifications.

As robots are increasingly deployed in open-world, human-centered environments, their ability to interpret natural language instructions and adhere to safety constraints is critical for effective and trustworthy interaction. Existing approaches often focus on mapping language to reward functions, rather than explicit safety specifications, or they address narrow constraint classes (e.g., obstacle avoidance), limiting their robustness and generality. We propose a modular framework for language-conditioned safety in robot navigation that enables free-form language instructions to define enforceable safety constraints. Our framework is composed of three core components: (1) a large language model (LLM)-based semantic parser that translates free-form instructions into structured safety specifications; (2) a perception module that grounds these specifications through object-level 3D scene representations; and (3) a model predictive control (MPC)-based safety filter that enforces both semantic and geometric constraints in real time, independent of the underlying task policy. We evaluate the effectiveness of the proposed framework through both simulation studies and hardware experiments, demonstrating that it robustly interprets and enforces diverse language-specified constraints across a wide range of environments, enabling a principled integration of natural language and safety-critical control.

AI copilots can substantially boost human performance through shared control, but excessive assistance can induce over-reliance and skill atrophy. This paper studies how an embodied AI agent can act as a coach that accelerates human motor-skill development. We argue that effective coaching requires strategic scaffolding and stepping back that are aligned with the learner's capability, allowing productive failures that drive learning. We formalize the interactive AI coaching process as a non-cooperative dynamic game in which the learner optimizes task performance while the coach targets the learner's independent competence. Building on this formalism, we develop a reinforcement learning framework combining adaptive shared control with probabilistic models of the coach's causal influence on skill evolution, enabling tractable training of coaching policies. A comprehensive user study (N=33) on first-person-view drone racing shows significant gains in human learning outcomes over state-of-the-art AI coaching baselines.

End-to-end autonomous driving planners may result in unreliable behaviors even when perception and reasoning appear correct. Planning-oriented models can generate trajectories that violate road boundaries, collide with surrounding agents, or ignore traffic rules. Even the most advanced driving agents based on vision-language-action (VLA) models hallucinate and react to objects that are not present in the environment. We propose RiskGuard, a planner-agnostic trajectory safety layer based on differentiable risk fields. The method represents heterogeneous constraints, including road boundaries, dynamic agents, and traffic rules, as a unified risk field. It evaluates predicted trajectories by aggregating risk along the trajectory and performs risk-guided trajectory revision when safety violations are detected. Closed-loop experiments in the CARLA simulator show that RiskGuard eliminates unsafe behaviors for planning-oriented systems, removing off-lane events, collisions, and traffic-rule violations while improving route completion. For VLA driving agents, the risk field provides a physically grounded signal that exposes hallucination-induced behaviors and supports reliable downstream decision making. These results suggest that trajectory-level risk reasoning provides an effective and interpretable safety layer for modern end-to-end autonomous driving systems.

Large scale pre-training on text and image data along with diverse robot demonstrations has helped Vision Language Action models (VLAs) to generalize to novel tasks, objects and scenes. However, these models are still susceptible to failure in the presence of execution-time impediments such as distractors and physical obstructions in the robot’s workspace. Existing policy improvement methods finetune base VLAs to improve generalization, yet they still struggle in unseen distractor settings. To address this problem, we investigate whether internet-scale pretraining of large vision-language models (VLMs) can be leveraged to reason about these impediments and mitigate policy failures. To this end, we propose StageCraft, a training-free approach to improve pretrained VLA policy performance by manipulating the environment’s initial state using VLM-based in-context reasoning. StageCraft takes policy rollout videos and success labels as input and leverages VLM’s reasoning ability to infer which objects in the initial state need to be manipulated to avoid anticipated execution failures. StageCraft is an extensible plug-and-play module that does not introduce additional constraints on the underlying policy, and only requires a few policy rollouts to work. We evaluate performance of state-of-the-art VLA models with StageCraft and show an absolute 40% performance improvement across three real world task domains involving diverse distractors and obstructions. Our simulation experiments in RLBench empirically show that StageCraft tailors its extent of intervention based on the strength of the underlying policy and improves its performance with more in-context samples.

Large Language models (LLMs) have shown promise as generators of symbolic control policies, producing interpretable program-like representations through iterative search. However, these models are not capable of separating the functional structure of a policy from the numerical values it is parametrized by, thus making the search process slow and inefficient. We propose a hybrid approach that decouples structural synthesis from parameter optimization by introducing an additional optimization layer for local parameter search. In our method, the numerical parameters of LLM-generated programs are extracted and optimized numerically to maximize task performance. With this integration, an LLM iterates over the functional structure of programs, while a separate optimization loop is used to find a locally optimal set of parameters accompanying candidate programs. We evaluate our method on a set of control tasks, showing that it achieves higher returns and up to an order of magnitude improved sample efficiency compared to purely LLM-guided search. We show that combining symbolic program synthesis with numerical optimization yields interpretable yet high-performing policies, bridging the gap between language-model-guided design and classical control tuning. Our code is available at https://sites.google.com/berkeley.edu/colmo.

Robots need task planning methods to generate action sequences for complex tasks. Recent work on adversarial attacks has revealed significant vulnerabilities in existing robot task planners, especially those built on foundation models. In this paper, we aim to address these security challenges by introducing PROTEA, an LLM-as-a-Judge defense mechanism, to evaluate the security of task plans. PROTEA is developed to address the dimensionality and history challenges in plan safety assessment. We used different LLMs to implement multiple versions of PROTEA for comparison purposes. For systematic evaluations, we created a dataset containing both benign and malicious task plans, where the harmful behaviors were injected at varying levels of stealthiness. Our results provide actionable insights for robotic system practitioners seeking to enhance robustness and security of their task planning systems.

A symbolic safety shield decides whether to permit a robot action by evaluating a logical constraint over predicates that a learned perception module grounds from sensor data. When the grounding is wrong the shield fails silently. Conformal prediction repairs this by replacing point estimates with prediction sets, and recent shield constructions do exactly that, but they assume a single prediction set over the whole state. Embodied symbolic states are factored: a tabletop with six objects already carries 90 grounded predicates, and the count grows quadratically with object count. Conformalising such a state either spends the risk budget through a union bound over all $N$ predicates or sets one joint threshold from the hardest predicate in the scene. Both are close to vacuous in practice. We observe that a shield never needs the symbolic state. It needs the truth value of one Boolean formula, and that formula's syntax says which predicates matter, in which polarity, and in which combinations. We turn the grounded unsafe condition into a scalar nonconformity score -- the minimum over its satisfied clauses of the maximum literal error inside the clause -- and calibrate a single threshold from it by split conformal prediction. The resulting shield (SCOPE) is exactly valid with no union bound and no dependence on $N$. On a tabletop domain with a convolutional predicate grounder whose per-predicate AUROC ranges from $0.811$ to $0.996$, SCOPE holds the unsafe-action admission rate at or below every requested budget while blocking $6.7%$ of safe actions at $\alpha = 0.10$, against $48.1%$ for a joint-threshold shield over the full state and $65.4%$ for the Bonferroni construction. We also report three negative results: normalising for per-predicate difficulty buys nothing once clause structure is used; at a matched admission rate the certified shield remains slightly more conservative than an uncalibrated plug-in threshold; and in closed-loop execution a policy that screens several candidates and takes the first permitted one exceeds the per-action budget, so the guarantee has to be restated over the candidate set before it applies to a running robot -- dividing the budget by the candidate count restores it at a cost of about five points of task completion.

Large Vision Models trained on internet-scale data have demonstrated strong capabilities in segmenting and semantically understanding object parts, even in cluttered scenes. However, while these models can direct a robot toward the general region of an object, they lack the geometric understanding required to precisely control dexterous robotic hands for 3D grasping. To overcome this, our key insight is to leverage simulation with a force-closure grasping generation pipeline that understands local geometries of the hand and object in the scene. Because this pipeline is slow and requires ground-truth observations, the generated dataset is distilled into a diffusion model that can operate on camera point clouds. By combining the global semantic understanding of internet-scale models with the geometric precision of a simulation-based locally-aware force-closure, IFG achieves high-performance semantic grasping without any manually collected training data. For visualizations, please visit our website at https://ifgrasping.github.io/

Technological advancements are pushing the boundaries of Industry 4.0's techno-centric development of smart machines into the new era of Industry 5.0, which aims to foster closer collaboration between the human workforce and these smart machines. However, one challenge remains problematic: human behavior. Human behavior is often unpredictable and nuanced, making it difficult for machines to observe, reason about, and work safely and efficiently alongside their human collaborators. In this paper, we introduce Onto-Task (OxT), a modular ontology for monitoring and correcting agentic behavior. Onto-Task provides a structured ontological framework that helps robotic agents assess a human collaborator's behavior in hybrid, unstructured environments and take corrective action based on these observations. This paper highlights the behavior pattern within Onto-Task, adapted from the ontology for the interpretation of human behavior (OntoPret), and presents it as the core contribution for modeling human behavior in human-robot collaborative (HRC) environments.

Norm learning from demonstrations typically assumes that the representation used to express norms is fixed in advance, even though the relevant representation may itself be uncertain. We formulate norm discovery as joint inference over a normative abstraction and a norm, using an LLM to propose executable abstraction--norm hypotheses and Bayesian maximum-entropy inference to evaluate how well they explain demonstrated behavior. When the current hypothesis set cannot adequately account for new observations, the learner can refine the representation by introducing higher-level state properties, actions, or events. We evaluate the approach in a deterministic shopping gridworld with existing social norms. GPT-5.5 achieves $0.93\pm0.14$ accuracy on classifying held-out behavior that requires generalization beyond the initial low-level representation, compared with $0.333$ for a fixed-representation MLCI baseline. These results support treating representation selection as an explicit part of norm discovery rather than assuming the appropriate normative representation is known in advance.

Robots that store past experiences must select which one to reuse in a new scene. Most systems select by visual similarity, and most evaluations report only the success of the selected experience. That number does not show whether the selection was good: a rule can score well by repeatedly using one broadly transferable experience, or poorly because its preferred experience is weak. This matters because robots increasingly adapt by reuse rather than retraining: every such adaptation passes through this selection step, and a score that describes the library rather than the rule misleads what the field builds next. We contribute an audit methodology: execute every stored experience in every query scene, yielding the complete table of transfer outcomes, over two manipulation tasks, three reuse mechanisms, and libraries of $K=3$, $10$, and $50$. The complete table is what makes the confound measurable: every alternative's outcome is known, so a score can be traced to per-scene selection or to library quality. The audited rules select by nearest-neighbor distance in five visual embeddings, from raw pixels to CLIP. We see four results. (1) One fixed experience, chosen with hindsight, captures 30--58% of the gap between random selection and an oracle; per-scene selection competes for the remaining 0.07--0.15 in success rate. (2) At $K\ge10$, visual rules concentrate on one experience 1.5--3 times more than the oracle does, and their scores then follow that experience's quality. (3) Wherever a rule differs significantly from a shuffle that keeps its selection rates but pairs them with scenes at random, the rule is worse, for every learned image policy. (4) Visual distance predicts well whether a given pair will succeed (AUROC up to 0.96), yet ranks the candidates within one scene no better than chance for four of five embeddings at $K=50$ (AUROC 0.45--0.52). Exhaustive execution is usually infeasible, so the audit reduces to two cheap reports any study can give: the distribution of selected experiences, and the success of the best single experience in hindsight.

Micro aerial vehicles (MAVs) are advancing toward operation in warehouses and other environments shared with humans, where safety assurance is a prerequisite for deployment. Obstacle avoidance on MAVs, however, remains agnostic to semantics. Expert-chosen margin parameters, fixed before takeoff, apply equally to a person and a cardboard box. Vision-language models (VLMs) can provide the missing semantic understanding, but their judgments have no safety guarantee. We present a hybrid architecture whose learned and model-based modules meet at a single narrow interface, where the VLM's judgment enters a formal safety layer only as parameters. Told "go to the whiteboard," an onboard VLM identifies the target and assigns a safety score to every surrounding object. The perception pipeline converts each object into a keep-out ellipsoid whose margin is determined by its score within fixed bounds. A control-barrier-function filter then enforces these constraints on every command, and holds them fixed for the duration of the mission. Safety holds by construction, independently of the VLM. The filtered trajectory stays outside every keep-out for any scores the model assigns, so a semantic error can lengthen the flown path but cannot cause a collision with a perceived object. The entire pipeline runs onboard an 8 GB Jetson Orin Nano, and is validated in simulation and in real flight experiments.

Experience replay can provide strong aggregate retention during continual adaptation of pretrained vision language action (VLA) models, but aggregate metrics may conceal substantial differences among individual skills. We study how forgetting varies across previously acquired skills under matched subsequent adaptations. Using a RoboCasa pretrained GR00T N1.5 policy, we conduct a controlled evaluation of 12 two stage adaptation pairs, in which three retained skills are each followed by the same four incoming tasks. Mean forgetting differs substantially across the three retained skills, reaching 8.83%, 20.67%, and 25.17%, respectively, whereas the four incoming task means span only 2.45%. These results provide preliminary evidence that the identity of the retained skill is an important source of forgetting variation in continual VLA adaptation. Motivated by this observation, we further examine whether task specific forgetting can inform replay allocation under a fixed memory budget. Under constrained memory, nonuniform replay increases final average success rate from 78.9% to 82.6% and reduces negative backward transfer from 8.4% to 4.5%. Overall, our results reveal substantial task specific structure in continual VLA forgetting and suggest that such differences can provide useful information for designing replay strategies.