SpatialClaw addresses a core limitation of current vision‑language models: their inability to reason accurately about where objects are, how they relate, and how they move in three‑dimensional space. Instead of retraining the model, SpatialClaw changes the way the model interacts with perception tools by treating executable code as the action interface. This lets an agent call pre‑built Python primitives—depth estimation, segmentation, geometry utilities—and immediately inspect the results before deciding the next step.
The framework wraps a stateful Python kernel around a VLM. The kernel holds input frames, metadata, and a toolbox of perception functions. A five‑stage loop—plan, generate code, execute, collect feedback, submit answer—runs until the agent calls ReturnAnswer() or reaches a step limit. Because the kernel persists variables, the agent can compose tools, verify intermediate outputs (e.g., checking a mask), and revise its approach, something that fixed JSON‑tool calls or single‑pass scripts cannot do.
In experiments across 20 benchmarks SpatialClaw achieved an average accuracy of 59.9%, outperforming the prior spatial agent SpaceTools by 11.2 points and beating a no‑tool baseline by 6.5 points. Gains are largest on dynamic tasks that require chained geometric reasoning across frames or viewpoints, such as measuring the closest distance between objects in a video.
For practitioners, the solution is training‑free: you can plug SpatialClaw onto any existing VLM backbone (Qwen3.5/3.6, Gemma4 families, etc.) without collecting new data or fine‑tuning. The setup is a single command that clones the repo, installs dependencies, and runs a benchmark on a machine with a GPU. The agent writes plain Python cells, uses a static AST checker for safety, and leverages utilities like KD‑tree for closest‑point queries or dot products for direction tasks.
Adopting SpatialClaw gives teams immediate improvements in spatial question answering, robotics planning, multi‑view inspection, and 4D video analysis—all without the overhead of model retraining.
#AI #SpatialReasoning #VisionLanguage #VLM #NVIDIA #DeepLearning