Test a system that depends on kafka-connect-jdbc by bootstrapping the Apache Kafka ecosystem artifacts and PostgreSQL through testcontainers.

In a nutshell, as described on Confluent blog the kafka-connect-jdbc provides the following functionality:

It enables you to pull data (source) from a database into Kafka, and to push data (sink) from a Kafka topic to a database.

This post is a showcase on how to test the synchronization of the contents of a PostgreSQL table via kafka-connect-jdbc towards Apache Kafka.

The contents of the input PostgreSQL table are synced as AVRO messages towards Apache Kafka.

The showcased project testcontainers-kafka-connect makes use of docker containers (via testcontainers library) for showcasing the Confluent Kakfa Connect functionality in an automated test case.

The interactions from this proof of concept are described visually in the image below:

The proof of concept testcontainers-kafka-connect project can be used for making end to end system test cases with docker for architectures that rely on kafka-connect-jdbc for syncing content from a relational database (PostgreSQL is the database used in the aforementioned project).

The testcontainers library already offers a Kafka module for interacting with Apache Kafka, but there is not, at the moment, a testcontainers module for the whole Confluent environment (Confluent Schema Registry / Apache Kafka Connect container support is missing from the module previously mentioned).

The project testcontainers-kafka-connect contains custom implementations for testcontainers extensions corresponding for:

• Apache Zookeeper
• Apache Kafka
• Confluent Schema Registry
• Apache Kafka Connect

that can be used to reproduce a multi-container scenario that involves working with the aforementioned components of the Apache Kafka ecosystem.

As a side note, the containers used do not use the default ports exposed by default in the artifacts (e.g. : Apache Zookeeper 2181, Apache Kafka 9092, Confluent Schema Registry 8081, Apache Kafka Connect 8083), but rather free ports available on the test machine avoiding therefor possible conflicts with already running services on the test machine.

For the test environment the following containers will be started:

• Apache Zookeeper
• Apache Kafka
• Confluent Schema Registry
• Confluent Kafka Connect
• PostgreSQL

It is quite impressive to see how close a productive environment can be simulated in the test cases with the testcontainers library.

Demo test

Once the test environment is started, via a HTTP call performed with rest-assured a kafka-connect-jdbc connector will be registered for the bookmarks PostgreSQL table.

The kafka-connect-jdbc connector will afterwards then continously poll the bookmarks table and will sync its contents towards the findinpath.bookmarks Apache Kafka topic.

The demo verifies whether the dynamically inserted contents into the bookmarks Postgres table get successfully synced in AVRO format on the Apache Kafka topic findinpath.bookmarks in the same order as they were inserted.

Sample code

Checkout the github project sample project testcontainers-kafka-connect and try out the tests via

mvn clean test