Google Cloud Blog
Follow
Unifying public and private data: Scale knowledge graphs with Data Commons on Spanner
The Data Commons project aims to organize the world's information and make it universally accessible and useful by unifying fragmented public datasets from over 100 authoritative providers. Data Commons provides a knowledge graph that connects real-world things and their relationships, with over 400 billion data points structured using standardized Schema.org definitions. The platform offers data exploration tools, MCP tools, and cloud-based APIs to access and integrate the clean datasets, making it easier for businesses to connect their internal data with public reference data. Data Commons integrates public information across multiple domains, including agriculture, demographics, economy, environment, and health, unlocking powerful use cases such as analyzing national GDP trends and tracking local health equity. The platform has transitioned to a native graph model with Spanner Graph, which brings the convenience of a SQL-like interface and graph expressiveness to Spanner, with its high availability, horizontal scale-out, and native ISO/IEC 39075 Graph Query Language support. By adopting a multi-entity Spanner Graph schema, Data Commons represents entities as nodes and their domain links as dynamic graph edges, allowing for complex relationship queries directly within the database using GQL. The new architecture simplifies pipelines by removing the need for complex, pre-computed indices and enables incremental updates to specific datasets without refreshing the entire database. Data Commons also adopts a lean implementation of the Statistical Data and Metadata eXchange technical standard, providing a consistent approach for describing and exchanging statistical data along with descriptive statistical meta-information. The Data Commons Platform update adds support for the SDMX technical standard version 3.0, providing out-of-the-box integration with third-party tools for multi-dimensional datasets. By federating across the public knowledge graph and a private knowledge graph containing their own data, users can light up exciting new use cases while maintaining data isolation and ensuring no data duplication.