DEV Community
Follow
Introducción a los Data Lakes Parte 2
A serverless Data Lake on AWS is built upon five fundamental pillars: storage, processing, catalog, security, and exploitation. Amazon S3 serves as the core storage, holding both raw and processed data in a structured folder hierarchy. Key S3 configurations include versioning, lifecycle policies for cost optimization, encryption, and cross-region replication. AWS Glue is the processing engine, offering Glue Jobs for ETL, Glue Catalog as a centralized metastore, and Glue Crawlers for schema discovery.Amazon Athena provides a serverless query service using standard SQL directly on S3 data, with pay-per-query pricing and native integration with the Glue Catalog. AWS Lambda acts as an orchestrator, triggered by events like S3 file uploads to initiate processes. AWS CDK is used for defining this infrastructure as code, ensuring reproducibility and version control.The typical data flow involves data ingestion into S3 raw, triggering Lambda via S3 notifications, cataloging by Glue Crawlers, processing by Glue Jobs, storing in S3 processed, and analysis via Athena. Cost optimization strategies include using Parquet format, compression, intelligent partitioning, S3 Intelligent Tiering, and lifecycle policies. Security is managed through IAM roles, policies, and end-to-end encryption across S3, Glue, and Athena. The next post will focus on practical AWS Glue implementation, covering job optimization, testing, and CI/CD integration.