Synthetic Users Fail Where Diversity Matters Most

Teams use LLMs as synthetic users to simulate human survey responses for product and market decisions. No one knows when this substitution is valid versus dangerously wrong.

Method: Four models tested across two survey domains (General Social Survey and World Values Survey) failed in two systematic ways. First, no LLM beat even the strongest non-LLM baseline fit on held-out human data. Second, models over-determined demographics, treating identity as far more predictive of attitudes than it is among real people—present for nearly every question-group combination. On a segment-targeting task, models inflated between-segment gaps two to fourfold, directed teams to the wrong segment in half of U.S. cases and most cross-cultural cases, and manufactured segment splits that don't exist in real people.

Caveats: Tested demographic prompting and standard survey-simulation protocols. Other prompting strategies might perform differently.

Reflections: Can any prompting strategy eliminate the demographic over-determination bias, or is it baked into training data? · Do models trained specifically on survey data (rather than general-purpose LLMs) show the same failures? · At what sample size does real human data become more cost-effective than debugging synthetic-user systems?

AI assistance might homogenize creative output from diverse groups, and AI-simulated diversity might replace human contributors altogether. Both would collapse the collective diversity that drives innovation.

Method: In a preregistered metaphor experiment, L2 English writers contributed more collective diversity than L1 writers. AI ideation compressed collective diversity for everyone and made the L2 advantage undetectable. AI refinement preserved both diversity and the L2 advantage. Simulated writer pools using personas from real backgrounds, three model families, native-language prompting, and elevated sampling temperatures all fell below every human pool. Pushing models further induced diversity only through degenerate text. However, AI ideation raised individual writers' ratings, creating a private incentive versus collective good tension—except when L2 writers used their native language, which benefited both.

Caveats: Tested on creative metaphor tasks. Other creative domains may show different patterns.

Reflections: Does the individual-versus-collective tension persist in other creative tasks beyond metaphor generation? · Can training data interventions make LLMs better at simulating human diversity without degenerating? · How long does the diversity-preserving effect of AI refinement last in longitudinal creative workflows?

Earable systems assume continuous wear, but no one has measured how long, how often, and in which situations people actually wear personal listening devices in the wild.

Method: Survey of 330 adults plus multi-year Apple Health logs from 90 participants revealed mean daily use increased from 37 minutes in 2020 to 64 minutes in 2024. Listening was intermittent: no listening logged on 48% of participant-days in 2024. Sessions were fewer but longer on weekends. Only 4% of participant-weeks exceeded WHO safe-listening limits. Sensation seeking (particularly disinhibition) showed small to medium positive associations with self-reported use. Most participants avoided PLD use in social situations.

Caveats: Apple Health logs only. Android users or non-Apple headphone users may differ.

Reflections: Do the 48% no-listening days cluster around specific life events or contexts that earables could detect? · Can sensation-seeking traits predict which users will adopt always-on earable features versus reject them? · How does intermittent wear affect the reliability of health metrics derived from earable sensors?