Abstract
Task planning bridges high-level instructions and executable behavior in long-horizon manipulation, yet modern Vision–Language–Action (VLA) systems often leave this intermediate structure implicit. Existing chain-of-thought (CoT) planners also tend to rely on coarse task-level annotations or serialize long reasoning traces token by token. We present X-Planner, a planning front-end that addresses both the supervision and representation of embodied reasoning. Our planning data combine Ego, UMI, and teleoperation under a Task/Subtask/Action/Segment hierarchy with source-dependent annotation depth. Takeover-time annotations and human-designed failures supervise ongoing error recognition. On the model side, a shared VLM backbone exposes two event-structured plan forms: a discrete interface that emits interpretable event states and a latent interface that relays continuous CoT states across staggered Transformer depths through Staircase Decoding. A frozen latent-to-text reconstruction objective provides a semantic anchor for the latent representation. Offline two-step planning evaluation places X-Planner second among four evaluated models on both BERTScore-F1 and a judge-based Overall score. In real-robot experiments, the complete event-mode system reaches average Task Progress scores of 71.60 and 53.75 on reasoning and generalization suites, respectively, outperforming the evaluated baselines. These results characterize planning-text quality and downstream execution; controlled plan-form ablations and end-to-end latency measurements remain open.
Methodtwo interfaces
Given a high-level instruction, available multi-view observations and optional history, X-Planner builds an event-structured plan that conditions a downstream world-action model. A shared Qwen-series VLM grounds the instruction in the scene, decomposes the task and estimates progress. The same backbone exposes two plan forms: interpretable event states in event mode, or continuous reasoning states in unified mode.
The complete planning architecture, Staircase Decoder and long-horizon manipulation sequence.
Event-grounded supervision
Ego, UMI and teleoperation data are organized under a Task/Subtask/Action/Segment hierarchy, with annotation depth adapted to each source. Takeover-time annotations and human-designed failures supervise error recognition during execution, rather than only describing successful task completion.
Discrete events and execution memory
At startup, the planner generates an ordered Initial Plan. As observations arrive, it updates a structured event state containing normalized progress, the current event, the next event when available, continuation state and execution-error recognition. In the demonstrated event interface, Long Memory retains previous predicted events while Short Memory provides the preceding event context. These readable states support inspection and monitoring, and their event descriptions condition the downstream world-action model. The episode-end marker identifies the recorded-media boundary; it does not alone certify task success.
Continuous plans with Staircase Decoding
The latent interface replaces token-by-token plan serialization with a compact sequence of continuous reasoning states. Staircase Decoding shares visual-language grounding below a relay depth, then updates reasoning positions in parallel through specialized upper Transformer blocks with independent causal caches. Projected VLM text states and planning latents enter the downstream cross-attention pathway alongside image tokens, preserving the world-action model's conditioning interface.
A frozen semantic anchor
A prefix projector maps latent states into the embedding space of a lightweight, frozen language model, which reconstructs the corresponding textual CoT trace. Training updates the staircase branch and projector while keeping the reconstruction model fixed. This objective anchors latent plans to event semantics without copying a particular autoregressive hidden-state trajectory. The depth-parallel schedule reduces serial planner computation; end-to-end control latency remains an open measurement rather than an established result.
Planning DemonstrationsRoboDojo simulation
Offline X-Planner predictions on recorded episodes: initial plans, multi-view observations, action predictions and memory. Robot motion comes from the recordings.
Loading rollouts…
Results
Task Progress on two real-robot suites
Loading results…
XPlanner-Benchmark
XPlanner-Benchmark on Hugging Face
A deterministic evaluation collection with an audited source inventory, separated from the training holdout by stable episode identity and manifest SHA-256.
BibTeX
@misc{xplanner,
title = {X-Planner: Event-Structured Task Planning for Embodied Intelligence},
author = {{X Square Robot Team}},
howpublished = {\url{https://github.com/X-Square-Robot/Xplanner}}
}