Google Cloud Blog
Follow
Managed Apache Iceberg at scale: How Spanner powers Lakehouse runtime catalog
Customers are adopting lakehouse architectures, standardizing on open formats like Apache Iceberg to create shared data estates for AI-native solutions. A crucial component is the Apache Iceberg REST Catalog, responsible for managing table pointers, atomic commits, and serving as the single source of truth. Enterprises need a highly scalable, available, and managed catalog to support the massive scale of small, repeated queries from AI agents. Such a catalog must address challenges like atomic commits, high availability, database scaling, table maintenance, and robust governance. Google Cloud’s serverless Lakehouse runtime catalog, powered by Spanner, offers a solution to these challenges.This catalog is fully serverless, highly available, and unifies metadata, natively implementing the Apache Iceberg REST Catalog specification. Key benefits include open APIs, multi-engine interoperability (BigQuery, Managed Spark), and read/write capabilities for Iceberg tables. It provides fully managed Iceberg storage with enterprise-grade features and zero-data copy by pointing directly to existing data in object storage. The catalog also supports bi-directional federation across clouds (Databricks Unity, Snowflake Horizon, AWS Glue) and secure access using credential vending. It integrates with Knowledge Catalog and Cloud IAM for AI-powered context and governance, offering table-level security, search, and lineage.Crucially, it ensures atomic commits, concurrency control, high availability, and scalability, backed by Google's planet-scale infrastructure and Spanner. Spanner provides full relational SQL semantics with horizontal scale-out elasticity for reads and writes, achieving high availability through regional configurations. It automatically scales compute and storage, dynamically monitoring data volume and query load. This foundation enables serializable transactions, providing industry-leading consistency guarantees for agent-scale operations. The Lakehouse runtime catalog integrates with Knowledge Catalog for efficient discovery of Iceberg tables, leveraging Spanner's full-text and native vector search for better recall. This integrated approach removes operational overhead and accelerates time-to-market for AI agentic use cases. Google Cloud’s borderless Lakehouse unifies data, governance, and AI context, while reducing operational total cost of ownership.