Building Unified, Access-Aware Search in a Data Mesh

Over the last few years, I have worked on a distributed data platform in which each engineering group owns the data it produces from heterogeneous data processing systems. Our central platform follows a data mesh model, as every domain manages its own pipelines and storage, but discovery in the unified system should be seamless to anyone searching across the mesh. That goal sounded simple, but wasn’t that simple considering we host datasets from ultra and black undisclosed products. The first complaint we heard, repeatedly: