Synthetic Participants Fail Where It Counts

UX teams are replacing real user testing with GPT-simulated participants to cut costs and speed up iteration. The question is whether synthetic clicks predict real behavior well enough to guide design decisions.

Method: GPT failed to predict where humans click in 53% of first-click tests across twelve diverse UX studies (n=3431). Participant personas, chain-of-thought reasoning, and sampling parameter tweaks produced no meaningful fidelity improvements—they just inflated believability. The synthetic responses showed systematic distortions in both click patterns and cognitive reasoning that stem from LLMs' statistical nature and linguistic training data.

Caveats: Tested on first-click tests only. Other UX methods (surveys, interviews) may show different failure modes.

Reflections: Do LLM simulation failures extend to other behavioral methods like card sorting or tree testing? · Can hybrid approaches (real users for critical paths, synthetic for edge cases) avoid the worst distortions? · Are there specific task types where synthetic participants perform acceptably?

Policy debates fixate on recommendation algorithms as the lever for fixing social media. But platforms differ fundamentally in architecture—tree structures like Reddit versus complete graphs like TikTok—and that might matter more.

Method: Agent-based simulations across four platform architectures (tree, layered hierarchy, network, complete graph) and two algorithms (chronological, popularity-based) revealed that architecture dominates algorithm effects. On tree platforms like Reddit, the Hot algorithm had no detectable effect on information spread or quality. On complete structures like TikTok, the same algorithm triggered winner-take-all dynamics with strong negative effects on both spread and quality, making the quality-popularity relationship unpredictable.

Caveats: Agent-based simulation, not field data. Real platforms have hybrid architectures and more complex algorithms.

Reflections: Can architectural constraints be retrofitted onto existing platforms without destroying network effects? · Do hybrid architectures (e.g., subreddit trees within a network) inherit the worst or best properties of each? · How do user adaptation strategies differ across architectures over time?

Multi-agent LLM workflows outperform single prompts but fail in opaque ways. Errors in intermediate nodes propagate downstream, forcing developers to inspect long traces without knowing which agent to fix.

Method: PROTEA executes workflows, scores intermediate node outputs with configurable rubrics, and overlays per-node states on the workflow graph to localize bottlenecks. For systems with only final-answer supervision, it performs backward node evaluation: generating candidate node-level expectations from final-answer references, then comparing them with observed outputs. In two production-adjacent workflows, PROTEA improved document-inspection accuracy from 64.3% to 83.9% and recommendation Hit@5 from 0.30 to 0.38.

Caveats: Tested on two workflows. Generalization to other multi-agent architectures unverified.

Reflections: Does backward evaluation work when final-answer references are noisy or incomplete? · Can the rubric-scoring approach scale to workflows with dozens of nodes? · How do developers decide which node to fix when multiple nodes show low scores?