Connecting RDBs and Search Engines — Chapter 3

This chapter details verifying the end-to-end flow of change data from PostgreSQL to Kafka using Debezium. It starts by setting up the environment using Docker Compose, ensuring PostgreSQL, Kafka, Debezium Connect, and ZooKeeper are running. The setup includes creating a table in PostgreSQL, configuring the database user, and creating a publication for Debezium. The PostgreSQL container is configured to enable CDC by adjusting WAL settings. Next, the Debezium connector is registered, specifying database details and topic configurations. The connector's status is checked to confirm it is running, and Kafka topics are listed to verify the connector's operation. CDC events are then viewed from Kafka using a consumer tool. Live data changes are confirmed by inserting a new row in PostgreSQL and observing the corresponding Kafka output. The chapter explains the Debezium JSON event format, highlighting the meaning of "r," "c," "u," and "d" operation codes. Finally, the chapter provides troubleshooting steps for common issues like missing topics or events. The next chapter will involve using Flink to process this data.