Data Projects: Managing Data A... Note

Data Projects: Managing Data Assets at Netflix Scale

Netflix faced challenges managing millions of data assets with an individual asset-based access control system. Organizational changes caused permission updates to be burdensome and inconsistent, leading to either flooded support teams or overly broad access grants. Workloads were tied to human identities, causing failures when engineers changed roles or left the company, resulting in a "permissions whack-a-mole" issue. To address these problems, Netflix introduced Data Projects, which act as containers for related assets and provide durable, project-specific identities for workloads. This shifts management granularity from individual assets to logical project groupings. Each Data Project has grants and roles, allowing users, groups, or applications to be assigned specific access levels. The project’s Netflix application identity executes workloads, independent of human lifecycles, and can be cryptographically exchanged for AWS IAM roles. A key feature called "gravity" automatically adds newly created assets to their project, ensuring organizational consistency. Data Projects have proven successful in stabilizing critical pipelines and simplifying access management for complex analytics domains. Onboarding involves creating a project, granting roles, and configuring workflows to use the project's identity, with new assets automatically associated via gravity. Future plans include expanding projects beyond data to software and studio assets, integrating rightsizing for permissions, and adding features like cost attribution and auditing. Ultimately, Data Projects provide a scalable and stable solution for identity and access management at Netflix, aligning with how teams organize their work.
CdXz5zHNQW_K0v8hwjFH4.png