The Data Engineering Agent is now in preview

Google Cloud has introduced a Data Engineering Agent in BigQuery, powered by Gemini, to automate complex data engineering tasks. This agent aims to free up data professionals from repetitive work by enabling natural language pipeline creation and intelligent modification. It integrates with Dataplex Universal Catalog to enhance pipeline relevance and performance by leveraging metadata. The agent automates code documentation and can incorporate custom business logic through UDFs. It significantly simplifies data preparation, transformation, and modeling, including complex tasks like SCD Type 2 dimension creation. For troubleshooting, the agent analyzes logs and suggests solutions, reducing downtime. It also accelerates pipeline migrations, as demonstrated by Vodafone's success in reducing migration time by 90%. Early users like PRISA Media have found it valuable for automating maintenance and complex data modeling. The Data Engineering Agent is accessible now in BigQuery Studio or the Dataform UI. Future enhancements will include proactive troubleshooting, IDE integration, and pipeline orchestration.