UX Collective | Medium
Follow
I handed a UX review over to AI. Here’s what happened.
AI can be a valuable partner for UX reviews but does not replace human expertise. Testing with Claude Fable 5 and ChatGPT-5.6 Sol for a UX review revealed that AI models produced analyses with irrelevant or fabricated issues. While both offered some useful observations, these were mainly beneficial for refining wording and structure. Claude demonstrated better accuracy in phrasing and structure compared to ChatGPT. The AI models struggled to identify all existing problems and missed several significant ones. A study on GPT-4o's ability to evaluate usability heuristics echoed these findings, showing it identified only a fraction of expert-identified problems. The author found that integrating personal observations and ongoing conversation with AI significantly improved the final analysis. Doing so sped up the process by an estimated 15% compared to a purely manual approach. The core limitation of AI in such tasks lies in its judgment and contextual understanding, not its access to data. Future approaches involve building custom AI agents that encode established heuristics and expert judgment for consistent application. The accuracy of AI evaluations stems from the quality of expert knowledge fed into the models, not the models themselves. While AI is expected to eventually automate many tasks, complex design and research roles will likely benefit from AI augmentation for some time. The author advises using AI as a preliminary tool, feeding it real data, and working in structured steps. It is crucial to avoid treating AI output as final or expecting a single prompt to replace a comprehensive workflow. Designers who learn to effectively direct AI will gain a significant advantage, and panicking about immediate job displacement is unnecessary.