Connecting RDBs and Search Engines — Chapter 5

This chapter details building a CDC pipeline using Flink SQL to join data from PostgreSQL. The architecture involves PostgreSQL, Debezium, Kafka, Flink SQL, and OpenSearch. It begins by setting up PostgreSQL tables for products and orders, including initial data and necessary permissions. A Debezium connector is registered in Kafka Connect to stream changes. An OpenSearch index is created with a specific mapping for storing joined data. Flink SQL is then employed to define Kafka tables for orders and products, using the Debezium-JSON format. An OpenSearch sink is configured to receive the joined data, and a view is created to join order and product information. An insert statement populates the OpenSearch index with the joined results. Finally, the Flink job is run and data validation steps are described, including verifying data in Kafka topics and OpenSearch using curl commands and a custom script. The chapter concludes by mentioning upcoming topics like deduplication and partitioning.