System One models in Robotics
Published:
Overview
The release of Jev by Typesafe AI started a revolution of structured output unstructured input system one models that can evaluate a state under multiple questions in parallel. This brings unprecedented speed and calibration in decision making. Robotics has multiple applications for such models. This work is under progress.
Research Ideas under exploration
Fast multi-modal action selection
Recent vision-language-action (VLA) models such as RT-2 and OpenVLA formulate robot control through tokenized action prediction, inheriting the autoregressive interface of large language models [Brohan et.al. 2023 RT2, Kim et.al. 2024 OpenVLA]. While this enables a unified interface for language and action, sequential token generation may introduce unnecessary latency when the robot only needs to select a simple low-level action. Inspired by recent System-One models such as Jev, which directly map unstructured inputs to typed probabilistic decisions [Almeida et.al. 2026 Jev], we propose to convert a pretrained multimodal Qwen model [Bai et.al. 2025 Qwen3] into a direct action-selection policy.
At each control step, the model receives the latest visual observation, proprioceptive state, and task instruction, and predicts \(\displaylines{ p_\theta(a \mid o_t), \qquad a \in \mathcal{A} }\) where $\mathcal{A}$ is a discrete low-level action dictionary containing commands such as $\pm1$,mm Cartesian translations, $\pm3^\circ$ gripper rotations, gripper open/close, and stop. We will also study a multi-resolution dictionary containing both coarse and fine motions. Instead of auto regressively generating action tokens, all candidate actions are scored in parallel using a lightweight classification or action-embedding head on top of the multimodal representation.
Jev-Style Independent Execution Supervisor
Most VLA policies only use as input, a short horizon of environment observations to predict actions. In such a case, generalist VLA policies might lose sight of the operation at large. It may repeatedly execute an action that is no longer appropriate because the target has already been reached, the robot has deviated from the intended motion, or an object has entered a collision-risk configuration due to only minor changes in environment observations. Moreover, VLA policies that implicitly undertake this at execution time run much slower than the control loop itself. Inspired by Jev’s formulation of typed probabilistic decisions [Almeida et.al. 2026 Jev], we propose an independent multimodal supervisor that monitors an executing command and decides whether it should continue or terminate, manage environment affordances, and maintain collision risk.
The supervisor receives the current visual observation, proprioceptive state, task instruction, and a short history of the executed action and observations. Rather than generating actions, it produces a categorical decision such as CONTINUE, STOP-SUCCESS, or STOP-UNSAFE in parallel. The decision is made only after a command has been executed for a short horizon, allowing the supervisor to assess whether the observed state is consistent with the intended outcome and whether continued execution is safe. In simulation, the supervisor can be trained using simulator state to automatically label execution states according to task completion, collision, and other termination conditions, avoiding additional human annotation. The main VLA policy remains unchanged, making the supervisor an independent safety and execution-monitoring module.
The central hypothesis is that separating action generation from execution verification can improve robustness without requiring the policy itself to learn all failure modes. The supervisor can terminate actions when an object is reached, when progress toward the commanded behavior stops, or when the robot enters a collision-prone state, while allowing normal execution to continue otherwise. We will compare the proposed supervisor against an unsupervised policy without monitoring, a confidence-based stopping rule, and a simple simulator-based collision checker. Evaluation will measure task success, collision rate, unnecessary early terminations, successful completion detection, and added inference latency. We will additionally evaluate robustness under object-position perturbations and execution deviations to determine whether the supervisor generalizes beyond the exact trajectories seen during training.
