Transformer model scales weather forecasting skill with minimal architecture changes

Weather forecasting is crucial for understanding and mitigating climate change, and data-driven approaches using deep learning have shown promise in improving accuracy. However, many methods use complex architectures without clear analysis of their success. Researchers introduced the Stormer transformer model, which achieves state-of-the-art performance with minimal changes to the standard transformer backbone. The key components of Stormer include weather-specific embedding, randomized dynamics forecast, and pressure-weighted loss. Stormer's core is a randomized forecasting objective that trains the model to forecast weather dynamics over varying time intervals, allowing it to produce multiple forecasts for better accuracy. The model performs well on short to medium-range forecasts and outperforms current methods beyond 7 days, requiring less training data and compute. Stormer's performance scales favorably with increases in model size and training tokens, making it a promising candidate for real-world deployment. The researchers provide a thorough analysis of Stormer's key components but do not address potential limitations or caveats, such as its performance on specific weather events or generalization to different regions. The introduction of Stormer represents a significant step forward in data-driven weather forecasting, demonstrating the potential of simple yet carefully designed transformer-based architectures. The model's favorable scaling properties and reduced computational requirements make it a promising candidate for further development in climate change mitigation and adaptation.