Synthetic Users vs. Human Respondents: Same Conclusion, Less Variation
This Nielsen Norman Group article examines the results of comparing AI-generated synthetic users to a survey of 604 human participants, finding that while both datasets reached the same overall conclusion about the dominance of the term 'UX' in describing a specific work process, synthetic responses exhibited significantly less variation in the terms used. The study highlights the limitations of synthetic-user platforms in capturing the full range of human responses and the potential impact of presentation format on respondent answers.
Original reporting · nngroup.com
Read “Synthetic Users vs. Human Respondents: Same Conclusion, Less Variation” at Nielsen Norman Group →How this was written. 1to1 wrote this article with a language model from Nielsen Norman Group’s reporting (model: @cf/ibm-granite/granite-4.0-h-micro); it describes that reporting rather than adding to it. The original, linked above, is the full story.
Key points from Nielsen Norman Group’s report
- Synthetic respondents and human respondents both concluded that 'UX' is the dominant term for the described work process, but synthetic responses showed far less variation in the terms used.
- Synthetic respondents used a narrower range of terminology, with only 14% employing UX-related labels compared to 70% of human respondents, indicating mode collapse in the synthetic data.
- The synthetic dataset was more limited in capturing less-common or emerging terms, as it produced none of the specialized labels like 'human factors' or 'experience design' that appeared in the human data.
- The synthetic respondents were generated using a platform that simulates interview-style responses, which may have influenced their answers and introduced a confounding variable, despite researchers' limited control over question presentation.
- While synthetic users offer a convenient alternative to recruiting real participants, the study suggests they may not fully represent the full spectrum of human responses, particularly in terms of uncovering less-common or emerging terminology and ideas for further research.
From the source
Data from AI-generated respondents pointed to the same conclusion as a survey of 604 humans but contained far fewer distinct answers.