Ed Crewe: Talk about Cloud Prices at PyConLT 2025

The author is speaking at PyConLT 2025 about cloud pricing and will discuss refactoring a data pipeline and pipeline framework options. The author works at EDB, the Postgres company, on their Postgres AI product, which runs across AWS, Azure, and GCP. The team handles identity management and billing components, including cloud prices ETL using Python and Airflow. Cloud pricing is complex, with almost 5 million prices (SKUs) across the three cloud providers. The team needs to gather all the latest service SKUs and total them up as a cost estimate for customers. To do this, they built a data pipeline to gather the SKUs and keep them current. The pipeline is in Apache Airflow and uses an abstract base class for scrapers to fetch cloud prices from each CSP. The pipeline also uses embedded Postgres and Click, which allows for fast stand-up and tear-down of temporary embedded storage. The author will discuss the benefits of this approach for development and testing. The final data artefacts will be loaded to a Postgres cluster price schema micro-service running on CloudNativePG.