RSS Google AI Blog
Follow
Testing LLMs on superconductivity research questions
AI is increasingly used in everyday tasks and holds significant potential for accelerating scientific research. A recent study investigated how well large language models (LLMs) answer expert-level questions in condensed matter physics, specifically high-temperature superconductors. The study, involving experts and multiple LLMs, focused on an open area of inquiry, the underlying mechanisms of superconductivity in cuprates. Researchers evaluated six LLMs, including GPT-4o, Perplexity, and Claude 3.5, assessing their responses on criteria like balance, comprehensiveness, and evidence. The analysis revealed that LLMs using curated, quality-controlled sources (NotebookLM and a custom system) outperformed those using unfiltered internet data. These top-performing models exhibited strengths in providing balanced perspectives and comprehensive answers. The study also identified areas for improvement, like temporal understanding and visual reasoning, in all tested systems. The findings highlight the importance of expert-curated data and inform the development of trustworthy AI tools for scientific discovery. A reliable AI research partner could assist scientists and students in efficiently navigating complex scientific literature. While LLMs show promise, the research underscores the continued need for expert evaluation in specialized fields. Overall, this effort aims to advance scientific progress through the development of better AI tools.