AI 'coding agents' told not to hit things kept hitting things anyway
A new preprint says robot-controlling language models ignore safety instructions almost every time — and offers a fix that still fails one job in three.
The claim
A preprint uploaded to arXiv this month makes an uncomfortable point about a fashionable technique in robotics: letting a language model write the actual control program for a robot arm, rather than relying on hand-built, robot-specific code. Researchers from the University of Southern California and Purdue set out to test whether these “coding agents” behave safely when given a simple rule — do the task, but don’t touch this obstacle. According to the paper, the agents ignored that rule and collided with the obstacle in most trials.
Worth flagging up front: this is a preprint. It has not been peer-reviewed, and the paper itself was only submitted in September 2026. That doesn’t make the findings wrong, but it does mean the results haven’t yet been independently checked or replicated.
What the paper actually tested
The setup is narrower than “robots are dangerous” headlines might suggest. The researchers gave a coding-agent-controlled robot arm manipulation tasks — pick something up, move something — each paired with an explicit obstacle it was told not to touch. They then watched what happened.
The interesting finding isn’t that the model failed to notice the obstacle or misunderstood the instruction. The paper says the model’s own reasoning traces show it acknowledging the obstacle and the rule against touching it. The failure, according to the authors, is in planning: the model never actually treats “don’t touch this” as something that outranks “finish the task.” It has no built-in concept of plotting a clear route around an object, and no mechanism to notice a planned path has become unsafe and change course. At the moment of actually gripping or placing an object, it also has no awareness that contact itself needs to stay clear of the obstacle.
The proposed fix, and its limits
The team’s answer is a system called SafeHarness, which adds two extra steps around the language model rather than retraining it. One draws candidate routes as waypoints around boxed-off obstacle zones, checks them, and replans if a route becomes blocked. The other specifically constrains where and how the robot makes contact with an object, so the grasping motion itself doesn’t clip the thing it’s meant to avoid.
The reported numbers: 71.9% task success and 87.5% collision avoidance with SafeHarness, against a prior best system’s roughly 65% and 60%. Framed differently, that’s the agent completing the task correctly a little under three-quarters of the time, and still colliding with the forbidden object in roughly one attempt in eight. That’s a real improvement on the baseline — but it is not “solved.”
So who is actually affected
Nobody with a household robot or a warehouse picker is at risk from this today. This is a research benchmark, run on coding-agent systems in controlled manipulation tasks, not a report on any deployed product. The relevance is for the robotics and AI research community currently excited about using general-purpose language models to write robot control code on the fly, skipping the traditional route of engineering robot-specific software.
The takeaway
The core, checkable claim here is plausible and specific: an AI system can be told a safety rule, appear to understand it in its own reasoning, and still fail to act on it because planning and instruction-following aren’t the same thing. That’s a useful caution for anyone building on this style of AI-controlled robotics. But it’s one unreviewed preprint, tested in a lab setting, proposing a partial fix that still leaves meaningful collision risk. File it as an early warning sign for a fast-moving research area, not evidence about robots you’ll actually encounter.