Python Software Foundation: Ho... Note

Python Software Foundation: How AWS Powers PyPI and the PSF

The Python Software Foundation (PSF) manages critical infrastructure like PyPI, Python.org, and CPython, supporting Python's widespread use across various sectors. Most of the PSF's infrastructure runs on AWS, with significant costs covered by AWS's Open Source Credits Program. PyPI handles billions of requests daily, with Fastly's edge caching reducing the need to hit AWS origin servers. However, uncached interactions like user uploads are managed by services like EC2, RDS, and S3. This year, AWS spending has notably increased by 69%, a departure from eight years of stable costs. This rise is attributed to increased usage from more users, automated agents installing packages, and numerous CI runs. The PSF emphasizes efficient credit utilization, advocating for upstream fixes in popular tools like GitHub Actions. A major ongoing project is migrating the entire fleet to Amazon EKS for better autoscaling. Currently, infrastructure work is heavily reliant on a small team, highlighting a critical need for more engineering staff. The PSF is seeking funding through PyPI Organization accounts and service agreements to enhance its services and mitigate supply chain risks.