Agents Finish Tasks but Leave People Unable to Use the Result

People read only the final reply from AI agents that do multi-step work. Current tests check whether tasks finished, not whether the person can understand and use the reply.

Method: Researchers built a reader model that takes the requester's position and checks whether a reply can be understood from what the requester has seen, uses familiar language, and answers the request. Five annotators agreed with it at a level within their own range. Across 480 replies from eight models, 54% failed. Failures mostly cited material the requester never saw or labels the agent invented. Asking for explicit requirements lowered the failure rate from 0.52 to 0.43. A second model rewriting the reply lowered it to 0.36.

Caveats: Tested on eight models from seven families with multi-step tasks. Results for other tasks and for different readers are not established.

Reflections: Does the reader model's judgment hold for non-expert requesters who read replies differently? · Can agents learn to avoid coining private labels without a rewriting step?

Safety tests check whether a chatbot offers crisis help to a user who discloses suicide thoughts. They do not check what happens when the user says no.

Method: Researchers ran 2,550 scripted conversations with chatbots. Each user disclosed suicide thoughts at three severity levels, refused each referral up to 10 times, then changed the subject. Nearly all chatbots gave a referral in the first reply (2,542 of 2,550). Persistence varied widely, from 0.01 to 9.94 refusals across chatbots. Newer versions of all five tested chatbots persisted less than older versions. Only 173 of 8,818 refusal-phase replies asked about what blocked the user from seeking help.

Caveats: Scripted conversations, not real users, and tested on a single set of chatbots at one point in time.

Reflections: Does persistence after refusal change outcomes for real users in crisis? · Why do newer model versions persist less, and is that a training choice?

When a computer-use agent drifts, users must work out what changed, what should have stayed untouched, and what can be recovered. Plan-based interfaces leave them supervising steps, not results.

Method: TaskVM represents each task as an editable contract. The contract lists desired outcomes, facts to preserve, and the live application state. User edits run as real interface actions. An independent check verifies the outcome. Each action is marked reversible, compensatable, or irreversible. Across a benchmark, a user study, and a live-model exploration, the contract approach improved contract compliance and users' recovery control and trust. Execution outcomes stayed the same.

Caveats: Results come from a benchmark, one controlled user study, and a live-model exploration. Sample sizes for the user study are not given in the summary.

Reflections: Do users keep editing contracts over long, multi-day agent tasks? · How accurate must the independent verification be before users rely on it?