UX Collective | Medium
Follow
Synthetic users are making me rethink “imperfect research”
The author previously believed that any research, even if imperfect, was better than none. However, the rise of synthetic users has caused them to reconsider this principle. Imperfect research previously involved some direct interaction with real users, despite limitations like small sample sizes or rough prototypes. These studies still provided genuine insights from actual people. Synthetic users, on the other hand, generate simulated responses based on AI models and data, not direct human experience. Teams can create synthetic participants based on their own assumptions, then treat the generated responses as validated research findings. This creates a dangerous false confidence, making unproven assumptions appear to be concrete insights. Unlike truly absent research, synthetic findings can disguise assumptions as evidence. Recent studies suggest that even advanced "digital twins" can be inaccurate and lead to misinterpretations, potentially altering product decisions. Giving synthetic users more data did not consistently improve accuracy in one study. Another study found that synthetic twins correlated only weakly with real human responses. The author now believes synthetic users are not equivalent to imperfect research due to their fundamentally different risks. A potential valid use for synthetic users is as a practice tool for researchers before engaging with real participants. Ultimately, imperfect research yields incomplete insights into real people, while synthetic research generates predicted responses. Treating synthetic output as research findings is a significant and concerning shift.