DEV Community
Follow
How GitHub Is Rebuilding Its Git Storage for the Age of AI Agents
GitHub is rebuilding its Git repository storage system due to the increased traffic from AI agents. AI agents commit and push code much more frequently than humans, overwhelming the existing storage infrastructure designed for human working rhythms. This surge in commits and pushes, driven by AI agents working in loops, has caused significant scaling issues. The current system, called Spokes, stores repositories on multiple file servers, with every copy participating in every write operation, slowing down the process for busy repositories. GitHub's new design separates data safety from traffic serving to address this bottleneck. It will minimize the part of a push that requires agreement among copies to only ref updates. By separating storage from compute, the authoritative copy will reside in Azure Blob Storage, while lightweight workers handle reads and caching. This architectural change allows for significantly higher write throughput, up to 35 times more in internal benchmarks. The core principles are to keep coordination points small and distinct from data serving. For individual developers, this means being mindful of agent commit frequency to avoid similar scaling problems in their own workflows. The fundamental difference between human and AI commit patterns necessitates this infrastructure evolution for software development.
https://dev.to/hamza140/how-github-is-rebuilding-its-git-storage-for-the-age-of-ai-agents-m13 dev.to