Your Model Choice Doesn't Matter Nearly as Much as You Think...And That's Actually Good News

Neuroscientists once scanned a dead salmon, finding "brain activity" due to random noise, highlighting the need for proper statistical controls. This mirrors modern machine learning, where improvements are often invalidated when baselines are properly implemented. Null models, which disregard input, achieve high scores on LLM benchmarks, indicating a focus on formatting over genuine understanding. Models often learn the "wrong things" like textures instead of shapes, leading to inaccurate evaluations despite high scores. "Embarrassingly simple" approaches, like linear regression, frequently outperform complex architectures by simply using the correct baseline. XGBoost, a 2016 algorithm, frequently wins in tabular data tasks, proving that data quality is more important than model architecture. Data quality, prompt engineering, and the use of strong baselines are crucial for successful AI development and are often overlooked in favor of complex architectures. Focusing on these aspects – data, prompting, retrieval, and evaluation – leads to more reliable and transferable results. Researchers must implement proper controls and baselines to avoid the pitfalls of celebrating "brain activity" in a metaphorical dead fish. The obsession with the latest models often overshadows the more important, lasting aspects of a successful AI project. The current trend prioritizes architectural innovation over essential aspects like robust data, proper prompt design, and reliable evaluation methods. Instead of the latest models, prioritizing data quality, prompt engineering, and strong baselines is key for achieving robust and transferable results.