SkillWeave: Weaving Heterogeneous Demonstrations into Long-Horizon Manipulation Skills

Each sub-task policy is a diffusion policy conditioned on an external RGB view, a wrist RGB view and proprioception (end-effector pose, arm joint angles, hand joint angles). Both views are encoded by a ResNet-18; the visual features are concatenated with proprioception to condition a 1-D conditional U-Net that denoises an action chunk.

Kinesthetic demonstrations put the demonstrator's hand and arm in frame during collection, but they are absent during autonomous execution. Rather than inpainting the demonstrator away, we keep the original RGB observations and condition visual feature aggregation on a mask of the manipulated object: features pooled inside the mask form a query, and cross-attention pools the spatial feature map against it. The attention distribution is additionally supervised by the normalized object mask. During training, masks come from SAM 3 offline; at deployment, a lightweight per-view U-Net predicts them directly from RGB, so neither SAM 3 nor inpainting runs online.

Independently trained policies do not automatically compose: small prediction errors accumulate, so the predecessor's terminal state drifts away from the states the successor was trained to start from. We leave both policies unchanged and instead modify how the predecessor finishes.

For each successor policy, the first states of its training demonstrations form an empirical start-state set (wrist pose, arm joints, hand joints, each normalized). A state's compatibility is its mean distance to its K nearest neighbours in that set. Once the sub-task enters its terminal phase, the predecessor samples L candidate action chunks per replanning step, predicts each endpoint, and executes the candidate with the smallest distance, repeating until the score drops below a threshold or a maximum steering horizon is reached. Because every candidate is sampled from the predecessor itself, the system stays within behaviour the predecessor supports while moving toward states familiar to the successor.

Demonstrations are collected on a 7-DoF xArm7 with a 16-DoF LEAP Hand, observed by a third-person RGB camera and a wrist-mounted RGB camera facing the palm. For each sub-task we collect 100 demonstrations with the modality that fits it. During teleoperation the operator drives the arm with a 6-DoF SpaceMouse while finger motion is captured by a camera, tracked with MediaPipe and retargeted to the robot hand by joint-angle mapping. During kinesthetic teaching the arm runs in drag/teach mode while the operator positions the fingers by hand under a compliant controller that yields to manual adjustment and holds the resulting pose. Synchronized RGB observations and proprioception (6D end-effector pose, 7D arm joints, 16D hand joints) are recorded at 30 Hz, with object positions randomized across episodes.

• Teleoperation averages 64% on arm-level reaching, reorientation and placement sub-tasks but only 5% on the dexterous ones, where retargeting and indirect contact feedback break down.
• On dexterous sub-tasks, raw kinesthetic observations reach 10% and KineDex's segment-and-inpaint reaches 20%, while mask conditioning reaches 65%: explicit object-centric conditioning beats reconstructing occluded pixels.
• The same sub-task policies composed naively average 5% end-to-end; with successor-aware terminal steering they average 27%. Strong isolated sub-tasks do not imply reliable long-horizon execution.