AI Assistants Need Limits Because Users Miss Their Errors

Trust research asks whether AI bends to users. This study asks the reverse: do users bend toward a confident AI that is wrong?

Method: 32 participants learned science topics from five chatbots built on GPT-5-mini. Each bot's briefing had 0, 25, 50, 75, or 100% of facts swapped for plausible falsehoods, all stated with full confidence. Planted falsehoods were chosen on 60% of falsified quiz items, versus 11% of correctly taught ones. Once absorbed, wrong answers were held at high confidence (8.4 out of 10). Confidence stopped separating right answers from wrong ones.

Caveats: Small study of 32 people on five science topics. Results may differ for other subjects or for people with prior expertise.

Reflections: Can claim-level reliability labels restore users' ability to tell right from wrong? · Does the absorption effect hold for topics users already know well?

Search-based AI tools give synthesized answers instead of link lists. Attackers can fill empty search topics with their own pages and get the AI to cite them.

Method: Researchers placed attacker-written pages into search topics with little reliable content, called data voids. They varied user prompt tone (skeptical, excited, neutral), three AI models (ChatGPT, Gemini, Sonar Pro), and how visible the pages were. Attacker content appeared in responses 31% of the time. A single attacker-controlled page was enough to succeed. Skeptical prompts only slightly reduced success.

Caveats: Measured across three specific AI tools and tested prompts. Smaller effects of prompt tone or model choice cannot be ruled out.

Reflections: Which defenses can detect a data void before the AI cites a page from it? · How much does a page's domain reputation change how often it gets cited?

VR input relies on mid-air gestures or controllers, which tire users during long work sessions. Phone-style surface tapping would be easier, but VR cannot yet tell which finger touched.

Method: TapID uses a flexible strap with an inertial motion sensor on each side of the wrist. Its signals are combined with headset hand tracking to detect surface taps and identify the finger used. In seated tests, touch detection reached an F1 score (a 0-to-1 accuracy measure) of 0.997. Finger identification reached 0.93 for one person and 0.91 across users after 10 refinement taps.

Caveats: Tested only with seated users at a table. Standing use, walking, and other hand positions were not reported.

Reflections: Does finger identification hold up when users stand or move? · How much refinement is needed before a new user gets reliable finger identification?