find in path

Parse JSON in Trino

Mon, 05 Sep 2022 00:00:00 GMT

This post showcases a more complex scenario which makes use of the JSON functions made available in Trino.

Say you are tasked with retrieving the number of companies by creator from records which have the following structure

{
   "createdBy":2,
   "teams":[
      {
         "companyId":10,
         "teamId":1
      }
   ]
}

Let’s setup on the fly a simple scenario via Trino:

docker run --rm -d --name trino -p 8080:8080 trinodb/trino

For the purpose of this exercise, we can use the Memory connector:

trino> use memory.default;
USE
trino:default> CREATE TABLE test1 (v varchar);
CREATE TABLE
trino:default> INSERT INTO test1
VALUES
    '{"createdBy":2,"teams":[{"companyId":10,"teamId":101},{"companyId":10,"teamId":102}]}',
    '{"createdBy":2,"teams":[{"companyId":20,"teamId":203},{"companyId":30,"teamId":204}]}',
    '{"createdBy":3,"teams":[{"companyId":10,"teamId":103},{"companyId":40,"teamId":104}]}';

Now that we have the premises to test our scenario, we can proceed to resolve the problem at hand.

When dealing with JSON data, JSON functions and operators from Trino can come in handy:

SELECT
   json_query(v, 'strict $.createdBy') AS created_by,
   json_query(v, 'strict $.teams[*].companyId' WITH ARRAY WRAPPER) AS companies
FROM test1;

 created_by | companies
------------+-----------
 2          | [10,10]
 2          | [20,30]
 3          | [10,40]
(3 rows)

The output for the companies column is of type VARCHAR. We need to parse this content to an ARRAY in order to expand it via UNNEST.

SELECT created_by,
       company
FROM (
    SELECT created_by,
           cast(json_parse(companies) as array(integer)) as companies
    FROM (
         SELECT
             json_query(v, 'strict $.createdBy') AS created_by,
             json_query(v, 'strict $.teams[*].companyId' WITH ARRAY WRAPPER) AS companies
         FROM test1
    )
)
CROSS JOIN UNNEST(companies) AS t(company);

created_by | company
------------+---------
 2          |      10
 2          |      10
 2          |      20
 2          |      30
 3          |      10
 3          |      40

Now the counting problem becomes a simple grouping problem:

WITH input AS (SELECT
                 json_query(v, 'strict $.createdBy')                             AS created_by,
                 json_query(v, 'strict $.teams[*].companyId' WITH ARRAY WRAPPER) AS companies
               FROM test1),
     companies_by_creator AS (SELECT
                                created_by,
                                cast(json_parse(companies) as array(integer)) as companies
                              FROM input),
     creator_to_company AS (SELECT
                              created_by,
                              company
                            FROM companies_by_creator
                                   CROSS JOIN UNNEST(companies) AS t(company)),
     creator_to_company_count AS (SELECT
                                    created_by,
                                    COUNT(DISTINCT company) as companies
                                  FROM creator_to_company
                                  GROUP BY created_by)

SELECT *
FROM creator_to_company_count
ORDER BY created_by;

 created_by | companies
------------+-----------
 2          |         3
 3          |         2

The query has been rewritten with common table expressions for improving its readability.

Now that the demo is complete, the test environment can be stopped via:

docker stop trino

Data Mesh showcase with dbt-trino

Sun, 19 Sep 2021 00:00:00 GMT

Perform data transformations over a Data Mesh of several databases with dbt-trino adapter for dbt.

Introduction

One frequently asked question in the context of using dbt tool is:

Can I connect my dbt project to two databases?

(see the answered question on the dbt website).

tldr; dbt stands for transformation as in T within ELT pipelines, it doesn’t move data from source to a warehouse.

The creators of the dbt however have added support for handling such scenarios via dbt-presto adapter.

Trino is a fork of the popular presto high performance, distributed SQL query engine for big data. This SQL query engine offers a helping hand in performing SQL queries on top of a myriad of data sources. Trino supports talking to the common relational databases (Postgres, MySQL) and also to data sources that don’t support SQL (AWS S3, Apache Kafka, Apache Cassandra, etc.). Feel free to check the list of supported Trino connectors for more details.

By using Trino, there can be queried data from fully separated databases. This makes Trino the analytics engine for data mesh (quote from Starburst Data website).

dbt Trino Architecture Image taken from Trino Community Broadcast

Data Mesh

Data Mesh is a paradigm to the data engineering domain which provides an alternative to the common recipe of using a centralized, monolithic data warehouse.

The principles on which this paradigm is being founded are quoted below:

domain-oriented decentralization of data ownership and architecture
domain-oriented data served as a product
self-serve data infrastructure as a platform to enable autonomous, domain-oriented data teams
federated governance to enable ecosystems and interoperability.

`jaffle_shop` Data Mesh

jaffle_shop is a self-contained project showcasing the functionality of dbt data warehouse transformation tool.

In the context of the project jaffle_shop there are being used the domains:

customers
orders (customers make orders)
payments (each completed order has a corresponding payment)

Entity Relationship Diagram image taken from jaffle_shop project

In the tutorial jaffle_shop project, all the domain entities (customers, orders, payments) as well as the results of the dbt transformations are obviously located in tables within the same database.

This blog post provides the answer to the questions answered by the project jaffle_shop in a data mesh decentralized data warehouse context where each domain (customer, order, payment) is being stored in a separate database. The insights gained from the dbt transformations will be also saved in a separate database.

Demo

Check out the dbt_jaffle_shop_data_mesh project to try out the concepts described in this blog post.

The project is using docker to spin up the jaffle_shop Data Mesh environment and for performing the dbt transformations. No other local installation or configuration of a tool is required on the workstation used for testing this showcase.

`dbt-trino`

dbt stands for transformation as in T within ELT pipelines, it doesn’t move data from source to a warehouse. dbt shines at orchestrating the transformations performed inside a data warehouse.

Below is a quote taken from Trino: The Definitive Guide book, Chapter 1, Introducing Trino :

Data warehouse systems have created not only huge benefits for users but also a burden on organizations:

Running and maintaining the data warehouse is a large, expensive project.

Dedicated teams run and manage the data warehouse and the associated ETL processes.

Getting the data into the warehouse requires users to break through red tape and typically takes too much time.

Trino, on the other hand, can be used as a virtual data warehouse. It can be used to define your semantic layer by using one tool and standard ANSI SQL. Once all the databases are configured as data sources in Trino, you can query them. Trino pro‐ vides the necessary compute power to query the storage in the databases. Using SQL and the supported functions and operators, Trino can provide you the desired data straight from the source. There is no need to copy, move, or transform the data before you can use it for your analysis.

Thanks to the standard SQL support against all connected data sources, you can cre‐ ate the desired semantic layer for querying from tools and end users in a simpler fashion. And that layer can encompass all underlying data sources without the need to migrate any data. Trino can query the data at the source and storage level.

dbt-trino is a dbt community adapter which offers a possible solution in dealing with transformations performed over a virtually distributed data warehouse.

By using this adapter, dbt is still interacting with only one SQL query engine, but in case of Trino, this is a fast distributed SQL query engine for big data analytics that helps you explore your distributed data universe.

trino> show tables in insights.default;
Table
-------------------
customer_orders
customer_payments
dim_customers
fct_orders
order_payments
stg_customers
stg_orders
stg_payments
(8 rows)

Query 20210918_214153_00126_zb79t, FINISHED, 1 node
Splits: 19 total, 19 done (100.00%)
0.21 [8 rows, 240B] [37 rows/s, 1.09KB/s]

As showcased in the demo project, in the snippet above, note that the tables:

stg_customers
stg_orders
stg_payments

are actually views, virtual tables, whose contents are defined by queries that are actually performed on external databases via Trino.

In the model dim_customers.sql from the demo project:

with customers as (

    select * from {{ ref('stg_customers') }}

),

customer_orders as (

    select * from {{ ref('customer_orders') }}

),

customer_payments as (

    select * from {{ ref('customer_payments') }}

),

final as (

    select
        customers.customer_id,
        customer_orders.first_order,
        customer_orders.most_recent_order,
        customer_orders.number_of_orders,
        customer_payments.total_amount as customer_lifetime_value

    from customers

    left join customer_orders on customers.customer_id = customer_orders.customer_id

    left join customer_payments on customers.customer_id = customer_payments.customer_id

)

select * from final

the data corresponding to the domain models of the jaffle_shop project:

customer
order
payment

is being joined in order to provide insights across all the aforementioned domain models.

Conclusion

In case that this post made you interested, feel free to check out the dbt-trino dbt adapter.

Try out on your workstation dbt_jaffle_shop_data_mesh to get you a quick look&feel on how dbt and Trino can be used together effectively for performing transformations on top of data distributed across various silos.

Feedback is very much welcome.

Inline Trino integer ranges

Mon, 19 Apr 2021 00:00:00 GMT

This post shows a few ways to create inline integer ranges in Trino.

While playing with a new database it is quite useful to have the ability to easily create a table with a integer range content (1, 2, 3, 4 and so on …) on which can be executed SELECT statements.

Use the memory connector and fill the table

Obviously one way to have a table to play with would be to actually create a table and fill it with values.

A simple way to implement this scenario is by using the Trino memory connector.

To configure the Memory connector, create a catalog properties file etc/catalog/memory.properties with the following contents:

connector.name=memory
memory.max-data-per-node=128MB

Create a table used for test purposes:

CREATE TABLE memory.default.test (value INTEGER);

Fill the table with values:

INSERT INTO memory.default.test (value) (VALUES 1,2,3,4,5,6,7,8,9,10);

Now, the inserted values can be queried/summed/counted:

SELECT value FROM memory.default.test;

Now obviously, the main inconvenient of this method is that the memory connector needs to be configured on Trino before starting to play with the range values.

Use an inline table

SELECT * FROM (VALUES 1, 2, 3, 4, 5, 6, 7, 8, 9, 10) as t(value)

Consult the documentation for VALUES SQL statement for more details.

The main benefit of this method is that is quite straightforward and it does everything in one split.

As explained on Trino documentation

Tasks operate on splits, which are sections of a larger data set. Stages at the lowest level of a distributed query plan retrieve data via splits from connectors, and intermediate stages at a higher level of a distributed query plan retrieve data from other stages.

When Trino is scheduling a query, the coordinator queries a connector for a list of all splits that are available for a table. The coordinator keeps track of which machines are running which tasks, and what splits are being processed by which tasks.

Unnest a generated sequence

This method requires the least typing:

SELECT value FROM UNNEST(SEQUENCE(1,10)) AS t(value);

Consult the documentation for UNNEST and SEQUENCE for more details.

UNNEST can be used also for an arbitrary array of integers:

SELECT value FROM UNNEST(ARRAY[1,2,3,4,5,6,7,8,9,10]) AS t(value);

Inline the range via `UNION ALL`

This method is syntactically similar to the one use to inline a table:

SELECT 1 as value UNION ALL
SELECT 2  UNION ALL
SELECT 3  UNION ALL
SELECT 4  UNION ALL
SELECT 5  UNION ALL
SELECT 6  UNION ALL
SELECT 7  UNION ALL
SELECT 8  UNION ALL
SELECT 9  UNION ALL
SELECT 10;

The main drawback in terms of efficiency of this method is that for each UNION ALL it creates a new Trino split.

Inline the range via `WITH RECURSIVE` clause

By means of recursiveness there can be obtained a similar result as with UNION ALL :

WITH RECURSIVE t(value) AS (
   VALUES (1)
   UNION ALL
   SELECT value + 1 FROM t WHERE n < 10
)
SELECT value FROM t;

Consult WITH RECURSIVE documentation for more details.

Also in this case, the main drawback in terms of efficiency of this method is that for each UNION ALL it creates a new Trino split.

Feedback

This blog post serves as a collection of methods that can be used for obtaining without too much effort integer ranges in Trino.

There are obviously other methods to obtain this functionality. Feel free to suggest new ways of easily obtaining ranges.

Infer deletions in parent-child ELT staging data

Wed, 17 Feb 2021 00:00:00 GMT

This post concentrates on the enhancing ELT staging data for being able to infer entries which are missing from parent-child JSON staging data.

Suppose that a video platform where the users can make channel subscriptions has a data pipeline to its Data Warehouse system for exporting in case of changes the data about its users in near real-time.

Let us suppose for the sake of simplicity that the data exported towards the data warehouse follows this format:

user
  - id: NUMBER
  - username: VARCHAR
  - created_at: DATETIME
  - exported_at: DATETIME

  subscriptions
    - id: NUMBER
    - name: VARCHAR
    - subscribed_at: DATETIME

Between the user and the subscriptions there is a 1:N relationship.

Obviously such a design is far from ideal in case of dealing with a real-world mainstream video platform where a user can have hundreds or even thousands of subscriptions, but due to the fact that video platform concepts are so spread, this is a nice fit for better explaining the problem tackled in this blog post.

In the time between two exports of the user subscription changes of a specific user data, there can be that new subscriptions get added and some of the subscriptions can also be removed (when the user unsubscribes from a channel).

e.g. : Let’s suppose that in the user subscriptions staging data there are found for the user johndoe two entries:

{
  "id": 1,
  "username": "johndoe",
  "created_at": 1609455600000,
  "exported_at": 1613379600000,
  "subscriptions": [
    {
      "id": 1000,
      "name": "5-Minute Crafts",
      "subscribed_at": 1612170000000
    },
    {
      "id": 2000,
      "name": "euronews",
      "subscribed_at": 1612256400000
    },
    {
      "id": 3000,
      "name": "Treehouse",
      "subscribed_at": 1612342800000
    }
  ]
}

and

{
  "id": 1,
  "username": "johndoe",
  "created_at": 1609455600000,
  "exported_at": 1613383200000,
  "subscriptions": [
    {
      "id": 2000,
      "name": "euronews",
      "subscribed_at": 1612256400000
    },
    {
      "id": 3000,
      "name": "Treehouse",
      "subscribed_at": 1612342800000
    },
    {
      "id": 4000,
      "name": "DevTips",
      "subscribed_at": 1613381400000
    }
  ]
}

Between 2021-02-15 09:00:00 and 2021-02-15 10:00:00 the user johndoe made a new subscription to the channel DevTips and unsubscribed from the channel 5-Minute Crafts.

This proof of concept project offers a solution on how to infer deletions of child entities within complex objects from the staging data.

In the scenario described above, an artificially created deletion entry should be created in order to point out approximately when the channel 5-Minute Crafts has been unsubscribed by the user johndoe.

On the basis of the insertions & deletions for the user subscriptions a data engineer can accurately find out across time the subscriptions that the users have made and how long they did last.

Historization of the user subscriptions

The basis for historization logic for the user subscriptions would be summarized by the union of the following data sets:

user subscriptions staging data
UNION
inferred deletions within the user subscriptions staging data (what this project concentrates on)
UNION
inferred deletions from the active user subscriptions within the already historized data
that do not appear in the last exported user subscription data

On the example of the johndoe user suggested above, the subscription availability intervals after processing the staging data at 2021-02-15 10:00:00 would be:

Channel name	Subscribed at	Unsubscribed at
5-Minute Crafts	`2021-02-01 09:00:00`	`2021-02-15 10:00:00`
euronews	`2021-02-02 09:00:00`
Treehouse	`2021-02-03 09:00:00`
DevTips	`2021-02-15 09:30:00`

Infer deletions in the staging data

Let us assume that the raw staging data looks exactly as in the hypothetical example presented previously for the user johndoe:

select *
from playground.dbt_video_platform.raw_user_subscription;

One possible solution for the problem described in the introduction of this project is presented in the code below:

WITH src_user_subscriptions AS (
        SELECT  load_id,
                exported_at,
                LEAD(load_id)  OVER (PARTITION BY user_id ORDER BY exported_at, load_id)      AS next_load_id,
                LEAD(exported_at)  OVER (PARTITION BY user_id ORDER BY exported_at, load_id)  AS next_exported_at,
                user_id,
                subscriptions,
                ARRAY_SIZE(subscriptions)                                                     AS subscriptions_size
        FROM (
                    SELECT id                                                    AS load_id,
                           record:"id"::NUMBER                                     AS user_id,
                           TO_TIMESTAMP(record:"exported_at"::NUMBER,3)            AS exported_at,
                           record:"subscriptions"::VARIANT                         AS subscriptions
                    FROM playground.dbt_video_platform.raw_user_subscription
        )
), user_subscription AS (
        SELECT src.load_id,
               src.exported_at,
               src.next_load_id,
               src.next_exported_at,
               src.user_id,
               subscription.value:"id"::NUMBER                              AS subscription_id,
               subscription.index                                           AS subscription_index,
               subscription.value:"name"::VARCHAR                           AS subscription_name,
               TO_TIMESTAMP(subscription.value:"subscribed_at"::NUMBER,3)   AS subscription_subscribed_at,
               src.subscriptions_size
        FROM src_user_subscriptions src
        LEFT OUTER JOIN TABLE(FLATTEN(input => src.subscriptions)) subscription ON 1=1
        WHERE subscriptions_size > 0
)

SELECT load_id,
       exported_at,
       user_id,
       subscription_id,
       subscription_index,
       subscription_name,
       subscription_subscribed_at,
       subscriptions_size,
       FALSE                                     AS _deleted
FROM user_subscription
WHERE subscriptions_size> 0

UNION ALL

-- create artificially deleted entries for the subscriptions which don't exist in
-- the next staging occurrence of the user's subscriptions
SELECT next_load_id                             AS load_id,
       next_exported_at                         AS exported_at,
       user_id,
       subscription_id,
       NULL                                     AS subscription_index,
       NULL                                     AS subscription_name,
       NULL                                     AS subscription_subscribed_at,
       NULL                                     AS subscriptions_size,
       TRUE                                     AS _deleted
FROM user_subscription   AS current_user_subscription
WHERE next_load_id IS NOT NULL
  AND NOT EXISTS (
        SELECT 1 FROM user_subscription AS next_user_subscription
        WHERE current_user_subscription.next_load_id = next_user_subscription.load_id
          AND current_user_subscription.subscription_id = next_user_subscription.subscription_id
  )

Summarized, the SQL query above takes the complex JSON user entities from the raw staging table and flatten them to user subscriptions. Subsequently each of the user subscription will get attached load_id and exported_at information from the next user subscription staging entry per user_id.

Inferring the deletions is now simple because there needs only to be checked whether a user subscription does not have a corresponding subscription in the next staging occurence for the user.

The outcome of this transformation can be seen below:

select *
from playground.dbt_video_platform.stg_user_subscription
order by load_id, subscription_id;

In the image above can be noticed that the subscription of the user johndoe with the ID 1000 identifying the 5-Minute Crafts channel that was present for the user in the staged entry with the ID 1 is being marked as deleted due to the fact that it does not occur anymore in the subsequent staging entry with the ID 2.

Demo

The concepts described in this blog post can be tried out in the project dbt_infer_deletions_in_parent_child_staging_data that is accompanying this blog post.

The project includes a data build tool model that can be used to infer deletions within the parent-child JSON ELT data. The nice thing about this SQL demo is that it contains also automatic test specifications (based on dtspec) in order to verify the accuracy of the implementation against Snowflake database.

Feedback

This blog post serves as a proof of concept for finding deletions within parent-child JSON staging data. Eventual improvements to the dbt_infer_deletions_in_parent_child_staging_data project code or ideas regarding alternative ways to solve this problem are very much welcome.

Infer deletions in batched ELT staging data

Sat, 06 Feb 2021 00:00:00 GMT

This post concentrates on the enhancing ELT staging data for being able to infer entries which are missing from one batch export to the next one.

Suppose that a shop is exporting daily towards the data warehouse environment the whole list of its active products in csv files:

products-2021-01-01.csv
products-2021-01-02.csv
products-2021-01-03.csv
products-2021-01-04.csv
products-2021-01-05.csv
…

Obviously from a day to the next there may be new products which get activated or existing products which get deactivated.

Nowadays there are source systems which simply do a snapshot of the existing products without providing any information about the deleted products since the previous export.

This missing information requires special effort on the data warehouse side in order to accurately infer the date range(s) for which the products in the shop were active.

If for a certain period of time, there is no product data ingested in the data warehouse, it may happen that the product data corresponding to several days in a row may need to be staged in the data warehouse.

This might be also the case when a bug is found in the historization of the product data which may involve a full reload of all the product staging data.

Historization of the product availability

In order to fully historize the product availability there needs to be made the following data sets need to be combined together:

product staging data
UNION
inferred deletions within the product staging data (what this blog post concentrates on)
UNION
inferred deletions from the active products within the already historized data
that do not appear in the last exported staging day

The formula briefly described above, provides the necessary input data for being able to historize in detail the product availability date ranges.

On the basis of the insertions & deletions for the products within the data warehouse environment the availability date ranges for each of the products from the shop can be accurately calculated.

Infer deletions in the staging data

This article doesn’t go in the details on how to historize the product availability within the data warehouse environment, but rather concentrates on how to enhance the staging data so to identify deleted entries within the staging data when exported data for the products corresponding to multiple days are imported at the same time.

If everything goes well and every day the products are staged successfully, the logic presented below will never be used.

On the other hand, when there are multiple days processed at the same time, it can get difficult to find out exactly which are the availability date ranges for a product which gets activated and deactivated several times during the processed days.

Let’s suppose that some of the products are very short-lived (e.g. : they are active only for a day on the shop).

e.g. : supposing that product data for each of the dates from the interval 2021-01-01 - 2021-01-05 is being staged at the same time. The product milk is available on 2021-01-01 , but not available anymore on 2021-01-02, then again available on 2021-01-04 and not available anymore again starting from 2021-01-05.

In the scenario described above, for the product milk, there should be created artificially deletion entries for the dates:

2021-01-02
2021-01-05

On the example of the milk product suggested above, the availability intervals in the staging data would be:

2021-01-01 - 2021-01-02
2021-01-04 - 2021-01-05

For the sake of an easier reading of what this (rather exotic) problem is trying to solve, here will be presented visually, step by step how the deletions of the products from the staged data are being inferred.

If the raw_products staging table has the following representation:

NOTE that here is made the supposition that the exported products for a certain day share the same export_time timestamp. The algorithm presented here works only when this supposition is met.

The starting point in the journey of inferring deletions within the staging data is to build a cartesian product between the staging dates and the staged product ids:

SELECT product_id,
       export_time
FROM (
        (SELECT DISTINCT product_id
        FROM playground.dbt_shop.raw_products)
        CROSS JOIN
        (SELECT DISTINCT export_time
        FROM playground.dbt_shop.raw_products) AS staged_date
);

From the cartesian product the following <product id, export_time> entries are chopped off:

the ones that happen before the first appearance of a staged entry for a product (marked in red)
the ones that are found within the staging entries (marked in yellow)

SELECT product_id,
       export_time
FROM (
          SELECT product_id,
                 export_time
          FROM (
                  (SELECT DISTINCT product_id
                  FROM playground.dbt_shop.raw_products)
                  CROSS JOIN
                  (SELECT DISTINCT export_time
                  FROM playground.dbt_shop.raw_products) AS staged_date
          )
) staged_product_date
WHERE staged_product_date.export_time > (
                                             SELECT MIN(export_time)
                                             FROM playground.dbt_shop.raw_products
                                             WHERE product_id = staged_product_date.product_id
                                         )
  AND staged_product_date.export_time
        NOT IN (
                    SELECT DISTINCT export_time
                    FROM playground.dbt_shop.raw_products
                    WHERE product_id = staged_product_date.product_id
               );

The result of this filtering is represented by all the <product, export_time> entries that are not found within the staging entries:

Now some of the resulting entries are not necessary. If a product is deleted after the first staging day and does not occur anymore subsequently, the entry with the minimum export_time is enough information.

What might happen though is that a product is deleted on one day, and appears again on a later day and is subsequently deleted again later. Between the two deletion dates we’ll have obviously a gap greater than one export day (because the product is in between at least one day active).

In order to cope with such a situation, the LAG window function is being used in order to identify greater gaps between the deletions :

SELECT product_id,
       export_time
FROM (
        SELECT product_id,
               export_time,
               LAG(export_time) OVER (PARTITION BY product_id ORDER BY export_time) prev_export_time
        FROM (
                  SELECT product_id,
                         export_time
                  FROM (
                          (SELECT DISTINCT product_id
                          FROM playground.dbt_shop.raw_products)
                          CROSS JOIN
                          (SELECT DISTINCT export_time
                          FROM playground.dbt_shop.raw_products) AS staged_date
                  )
        ) staged_product_date

        WHERE staged_product_date.export_time > (
                                                     SELECT MIN(export_time)
                                                     FROM playground.dbt_shop.raw_products
                                                     WHERE product_id = staged_product_date.product_id
                                                 )
          AND staged_product_date.export_time
                NOT IN (
                            SELECT DISTINCT export_time
                            FROM playground.dbt_shop.raw_products
                            WHERE product_id = staged_product_date.product_id
                       )
) AS missing_staged_product
WHERE (
        (missing_staged_product.prev_export_time IS NULL)
        OR missing_staged_product.export_time != (
                                                        SELECT MIN(export_time)
                                                        FROM playground.dbt_shop.raw_products
                                                        WHERE export_time > missing_staged_product.prev_export_time
                                                  )
);

The filtering can be observed (marked in red) in the following image:

The operation applied previously corresponds now to the inferred batched product deletions:

Simplified solution description

Instead of filtering out from the cartesian product of product_id and export_time to infer the deleted products, there could be used a much simpler solution.

It is assumed again that the raw_products staging table has the following representation:

The result of the query

SELECT
      export_time,
      LEAD(export_time) OVER
          (PARTITION BY product_id ORDER BY export_time)                   AS next_product_export_time,
      (
          SELECT MIN(export_time)
          FROM playground.dbt_shop.raw_products
          WHERE export_time > src.export_time
      )                                                                     AS next_export_time,
      product_id
FROM playground.dbt_shop.raw_products AS src;

can be used to see when the next occurrences of product exports happen in the staging data:

There can be now easily be seen that the next_product_export_time for some of the entries is greater that the next_export_time or that the next_product_export_time in some cases is simply NULL, otherwise said, is missing.

By applying the query

SELECT
    next_export_time               AS export_time,
    product_id,
    (
        next_export_time IS NOT NULL
        AND (next_product_export_time IS NULL OR next_product_export_time > next_export_time)
    )                              AS _deleted
FROM (
        SELECT
            export_time,
            LEAD(export_time) OVER
                (PARTITION BY product_id ORDER BY export_time)                   AS next_product_export_time,
            (
                SELECT MIN(export_time)
                FROM  playground.dbt_shop.raw_products
                WHERE export_time > src.export_time
            )                                                                     AS next_export_time,
            product_id
            FROM  playground.dbt_shop.raw_products AS src
)
WHERE _deleted

the valid entries:

the next_product_export_time is corresponding to the next_export_time (marked in red)
the next_export_time is NULL (marked in yellow) meaning that these are the product exports done on the last staging day

are being filtered out:

and the result is inferred from the product_id and next_export_time fields:

This second solution is definitely easier to grasp and also probably more efficient.

Demo

The concepts described in this blog post can be tried out in the project dbt_infer_deletions_in_batched_staging_data that is accompanying this blog post.

The project includes a data build tool model that can be used to infer deletions within the staging ELT data. The nice thing about this SQL demo is that it contains also automatic test specifications (based on dtspec) in order to verify the accuracy of the implementation against Snowflake database.

Feedback

This blog post serves as a proof of concept for finding deletions within staging data loaded in batched fashion. Eventual improvements to the dbt_infer_deletions_in_batched_staging_data project code or ideas regarding alternative ways to solve this problem are very much welcome.

dbt historized macro

Mon, 09 Nov 2020 00:00:00 GMT

Historize the status changes performed on the database entities an obtain the date ranges corresponding to the validity of each status in which the entity has been during its lifecycle.

Let’s use a fictional example of an online ecommerce store, where the orders table in the database backing the store is be one of the core entities.

An order can find itself, throughout its lifecycle in one of the statuses:

placed
shipped
completed
return_pending
returned

The content of the orders table could be ingested in a data warehouse environment in order to perform several reports concerning the performance of the ecommerce store.

Nowadays a key aspect in ecommerce is the delivery time, or more concretely in the case of our fictional ecommerce store, the period of time in which an order is marked with the status shipped (order has been shipped to the customer and is currently in transit).

The data warehouse environment could be therefore used to answer several questions that focus on the shipment:

What is the amount of orders that were marked in shipment during a certain day? If the trend of the orders in shipment doesn’t follow closely the trend of completed orders then the ecommerce shop has a growing number of unsatisfied customers.
How many orders took more than 3 business days during a certain month? Identify whether there the service level agreement for deliveries is not being respected.
What is the highest amount in days for an order in shipment during a certain month? From an operational perspective this number should be kept as low as possible. Possible spikes above the average should trigger automatic alerts for the person responsible on the deliveries.

For the sake of having a concrete example, let’s suppose that the structure of the staging table in the data warehouse environment that corresponds for the orders database table from the ecommerce store would have the following columns:

load_id: monotonically increasing sequence for the order changes
order_id: the order identifier
updated_at: the date when the order was updated
status: completion status of the order

Let’s try to answer the following question:

What is the amount of orders that were marked in shipment during a certain day?

NOTE that a shipment can take multiple days until the package reaches the customer and the order is marked as completed.

A naive answer would be to count the distinct orders from the staging table where status shipped is appearing during the specified day (e.g.: 2018-01-01)

SELECT COUNT(DISTINCT order_id)
FROM jaffle_shop.stg_orders
WHERE updated_at BETWEEN '2018-01-01' AND '2018-01-02'
AND status = 'shipped';

In this case the orders that started being shipped before the 2018-01-01 and completed during or after this day would not be taken into account.

One possible solution in order to the question previously mentioned would be to historize the status changes performed on the orders.

The image below contains historized order status changes. Note that the [valid_from, valid_to] date ranges corresponding to an order are adjacent to each other and only the latest entry corresponding to an order is unbounded (valid_to is set to NULL):

This technique allows to easily find the shipment date ranges that overlap with the day on which the number or orders in shipment needs to be calculated:

SELECT COUNT(DISTINCT(order_id))
FROM jaffle_shop.fct_orders
WHERE valid_from < '2018-01-02' AND (valid_to >= '2018-01-01' OR valid_to IS NULL)
AND status = 'shipped';

This post provides a proof of concept on how to historize order status changes with dbt models on Snowflake data warehouse environment.

Getting started with dbt

As described in the introduction to dbt :

dbt (data build tool) enables analytics engineers to transform data in their warehouses by simply writing select statements. dbt handles turning these select statements into tables and views.

dbt does the T in ELT (Extract, Load, Transform) processes – it doesn’t extract or load data, but it’s extremely good at transforming data that’s already loaded into your warehouse.

The jaffle_shop project is a useful minimum viable dbt project to get new dbt users up and running with their first dbt project. It includes seed files with generated data so that a user can run this project on their own warehouse.

For more information on dbt:

Read the introduction to dbt.
Read the dbt viewpoint.

Demo

This blog post is accompanied by the dbt_jaffle_shop_historized project which provides a working prototype for a minimal dbt project centered on order status changes.

The project contains the implementation for the historized dbt macro used for keeping track of the order status changes in the lifecycle of orders as well as corresponding dtspec (data transformation specification) tests used to ensure the accuracy for the dbt macro.

dbt historized macro

dbt offers the possibility of using macros. Macros are pieces of code that can be reused multiple times - same as functions in other programming languages.

By means of using a specialized macro for the historization functionality, the complexity of building the historization queries is fully abstracted and the readability of the query used for historizing the status changes is greatly simplified:

{{
    config(
        materialized = 'historized',
        primary_key_column_name = 'order_id',
        valid_from_column_name = 'updated_at',
        load_id_column_name = 'load_id'
    )
}}

select
        load_id,
        order_id,
        updated_at,
        status
from {{ ref('stg_orders') }}

As can be seen in the code snippet above, same as a function, the historized macro takes a few parameters:

primary_key_column_name: the name of the column used to identify the staged entity
valid_from_column_name: the name of the column containing the timestamp when the staged entity has been last updated.
load_id_column_name: the name of the column containing monotonically increasing sequence values used for distinguishing the precedence between order status changes that happen within the same timestamp.

Behind the scenes, the logic encapsulated in the historized dbt macro follows the steps mentioned below:

create a temporary table (suffixed by __tmp) which contains the staged entries along with md5_hash column containing hash of the concatenated columns used for historization (e.g. : status in case of this example, but nothing speaks agains historizing more columns on an entity). The hash column can be used to easily distinguish whether two adjacent entries (ordered by load id) are duplicated (have the same values for the versioned columns). For more information on hashing, read on The most underutilized function in SQL dbt blog post.
in case whether there are new entries staged (from the __tmp suffixed source table) that correspond to an unbounded historized entity (valid_to column is NULL) in the target table, then set the upper bound column valid_to to the value corresponding to the minimum valid_from value of the staged entries corresponding on the target entity
deduplicate the staged entries (based on the md5_hash column)
obtain the validity range for each of the staged records (valid_from, valid_to)
join the staged records with the records from the target table and filter out eventual duplicates (based on md5_hash)
insert the staged records in the target table.

For more information on dbt macros:

Read the introduction to Jinja macros.
Read about the dbt materializations and their corresponding implementation on Github.

Data tranformation testing

It takes quite a while to develop a dbt macro and afterwards it needs to be tested whether it works. It may even happen that while it is productively used, a bugfix needs to be done or that the functionality of the macro needs to be extended. By means of using automated tests for data transformations there could be ensured that the macro works as expected with an extensive battery of tests on a test environment.

dtspec is an open-source framework written in Python which can be used for specifying and testing data transformations.

Within dtspec is specified in a yaml format:

the source data in the table(s) to be used by the data transformation(dbt)
the expected data from the target table(s).

dtspec framework offers means to read the yaml specification, and match the data from the actual tables, once the data transformation has been performed (via dbt) with the data from the specification scenario.

Below is presented a snippet of the output used for running the tests on the project dbt_jaffle_shop_historized which accompanies this blog post:

Executing test specification tests/demo-spec.yml
Truncating data from the tables ['raw_orders', 'fct_orders']
Inserting input data into the source table raw_orders
/home/findinpath/dbt_jaffle_shop_historized/tests/..
Running with dbt=0.18.1
Found 2 models, 11 tests, 0 snapshots, 0 analyses, 148 macros, 0 operations, 1 seed file, 0 sources

21:51:29 | Concurrency: 4 threads (target='dev')
21:51:29 |
21:51:29 | 1 of 2 START table model jaffle_shop.stg_orders...................... [RUN]
21:51:31 | 1 of 2 OK created table model jaffle_shop.stg_orders................. [SUCCESS 1 in 2.02s]
21:51:31 | 2 of 2 START historized model jaffle_shop.fct_orders................. [RUN]
21:51:35 | 2 of 2 OK created historized model jaffle_shop.fct_orders............ [SUCCESS 1 in 4.39s]
21:51:36 |
21:51:36 | Finished running 1 table model, 1 historized model in 10.58s.

Completed successfully

Done. PASS=2 WARN=0 ERROR=0 SKIP=0 TOTAL=2
Loading data from the target table fct_orders
Loading actuals for target fct_orders
Asserting Building fct_orders out of the raw_orders when the fct_orders table is empty: Basic full refresh loading of the historized fact table `fct_orders`
Tested scenarios: - order1 : the expected historized entries should follow the representation: placed > shipped > completed - order2 : the expected historized entries should follow the representation: placed - order3 : contains duplicated (with the same `updated_at` and `status` values) entries. The duplicates should be filtered out from the processing. The expected historized entries should follow the representation: placed > shipped > return_pending
 PASSED

As can be seen from the log snippet above, there are several steps performed:

the source & target tables for the data transformation are truncated to make sure that the execution of the tests is fully repeatable.
input data is inserted in the source table(s)
dbt is executed for performing the data transformation on the: jaffle_shop.stg_orders, jaffle_shop.fct_orders models.
the actual data from target tables of the test specification is loaded and is used in asserting the equality with the content of the columns mentioned in the target table from the test specification

NOTE : When running the tests for this project, there can be noticed that it takes quite some time to execute them. This happens in part because Snowflake takes roughly a bit under a second to run any kind of query, even if the query is performed on an empty database.

Nevertheless, having automated tests for the dbt models, even though they are slow, is much better than doing the tests by hand or not doing them at all and leaving production as a playground for the developed models.

Feedback

This blog post serves as a proof of concept for historizing database entities with dbt. Eventual improvements to the dbt_jaffle_shop_historized project code or ideas regarding alternative ways to historize changes on the database entities are very much welcome.

Feature toggles as a service

Mon, 05 Oct 2020 00:00:00 GMT

A feature toggle is a technique in software development used for allowing to test a software feature on a live system even before it is completely implemented and ready to be released to the general public. The feature toggle is used to hide, enable or disable the feature during runtime.

Feature toggles are an important technique used for the implementation of continous delivery with the intricate aim of providing rapid feedback about the features implemented to the software engineers.

Feature toggles are essentially variables that are used inside conditional statements. Therefore, the blocks of code inside these conditional statements can be toggled ON or OFF depending on the value of the feature toggles. In a simple form a feature toggle works in the following fashion:

   if (modern page header is enabled) {
      // render modern page header
      // ...
   } else {
      // render classic page header
      // ...
   }

Feature toggles can be stored as:

properties in a configuration file
row entries in a database table
entries in external feature toggle service

Here is a description about feature flags functionality done by Chad Dickerson from Etsy:

“We practice continuous deployment and make small changes frequently to the site. We use what we call “config flags”, …and a lot of the code for features runs “dark” for days or weeks, and feature launches mean flipping a switch in the code.”

Ideally, the process of enabling or disabling a new feature in production should happen without any code changes, new deployment or application restart.

Unleash

As pointed out in the Unleash project page:

Feature toggles decouple deployment of code from release of new features

Through the usage of Unleash feature toggle service, new functionalities can be released to production with the possibility of enabling/disabling them at any later point of time through an admin UI outside of the running web application that makes use of the feature toggles.

Unleash feature toggle service comes with a lot of functionalities to address in depth the needs related to feature toggles:

Unleash client SDK can be embedded in any kind of application written in any popular programming language
provides an administration web UI to manage and track within the last minute/hour the usage of the feature toggles
provides a set of strategies associated with the feature toggles in order to enable gradual rollout of the features. In this way, a feature could be safely rolled out only for a subset (e.g. : specific emails, specific IPs, only Desktop/Mobile) of the users of the website that makes use of feature toggles.

Unleash also comes in an enterprise edition with additional features and a hosted option (SaaS). Check out https://unleash-hosted.com.

Common activation strategies offered by Unleash:

Active For users with a specified userId
GradualRollout to X-percent of the users
Active for beta users
Active only for application instances running on host x.

NOTE that Unleash offers the possibility to easily write custom strategies (e.g. : active for users with a specified device - based on the User-Agent web request header).

Below is shown an architectural overview of the Unleash system:

As can be seen from the picture above, the client applications are supposed to embed the Unleash client SDK and communicate with the server for reading the feature toggles. The Unleash server is written in Node.js.

At the time of this writing, there exist Unleash client SDK implementations for Java, Node.js, Go, Ruby, Python and .NET Core.

The client SDKs connect internally to Unleash REST API and retrieve the feature toggles with information about their associated strategy:

➜  ~ curl -s http://localhost:4242/api/client/features | jq
{
  "version": 1,
  "features": [
    {
      "name": "blueHeader",
      "description": "Whether the header should be shown in blue",
      "enabled": true,
      "strategies": [
        {
          "name": "default"
        }
      ],
      "variants": null,
      "createdAt": "2020-09-30T21:20:46.154Z"
    },
    {
      "name": "qotd",
      "description": "Quote of the day",
      "enabled": true,
      "strategies": [
        {
          "name": "userWithId",
          "parameters": {
            "userIds": "bob,john"
          }
        }
      ],
      "variants": null,
      "createdAt": "2020-09-30T21:34:37.612Z"
    },
    {
      "name": "specialGreeting",
      "description": "The user receives a customized special greeting",
      "enabled": true,
      "strategies": [
        {
          "name": "deviceClass",
          "parameters": {
            "device": "Desktop,Tablet"
          }
        }
      ],
      "variants": null,
      "createdAt": "2020-10-01T20:03:16.114Z"
    }
  ]
}

As pointed out by the Unleash documentation:

To be super fast, the client SDK caches all feature toggles and their current configuration in memory. The activation strategies are also implemented in the SDK.

This makes it really fast to check if a toggle is on or off because it is just a simple function operating on local state, without the need to poll data from the database.

NOTE that this means that no data (regardless whether it is sensitive or not) about the users (e.g. : username, IP, name, address, etc. ) of the website will be sent towards the Unleash server in order to check whether a feature is enabled when displaying a web page.

In case there exist doubts about having an ever increasing payload related to the Unleash feature toggles configuration that needs to be continously refreshed in the client application, Pete Hodgson, in his article Feature Toggles (aka Feature Flags), points out an interesting insight about the amount of feature toggles employed in a web application:

Savvy teams view their Feature Toggles as inventory which comes with a carrying cost, and work to keep that inventory as low as possible.

Competition for Unleash

Demo

This blog post comes with a proof of concept Spring Boot web application that showcases the functionality provided by Unleash feature toggle service.

Check out the source code on Github and feel free to try it out.

Articles related to feature toggles

Pete Hodgson - Feature Toggles (aka Feature Flags)
Martin Fowler - FeatureToggle

Feedback

This blog post serves as a proof of concept for using feature toggles. Eventual improvements to the project code or ideas regarding alternative ways to gradually roll out features are very much welcome.

Price range aggregations

Tue, 22 Sep 2020 00:00:00 GMT

Recipes for building dynamically balanced price range aggregations with Elasticsearch.

A common functionality in each webshop is the ability to show price range aggregations when searching for products.

When searching for a generic product (e.g. : backpack, shoes) the users are nowadays accustomed to see in the search result page a panel containing the distribution of the product prices matching their search.

e.g.: when searching for backpack on a marketplace website, there could be displayed the following price range listing:

up to 20 EUR - 10 items
20 to 40 EUR - 28 items
40 to 80 EUR - 52 items
80 to 180 EUR - 12 items
from 180 EUR - 4 items

Elasticsearch supports performing range aggregations on a search request, but it requires the user to specify explicitly the ranges for the aggregation:

curl -X GET "localhost:9200/products/_search?pretty" -H 'Content-Type: application/json' -d'
{
  "query": {
     "match": {
     	"category": "Luggage"
     }
  },
  "aggs": {
    "price_ranges": {
      "range": {
        "field": "price",
        "ranges": [
          { "to": 100.0 },
          { "from": 100.0, "to": 200.0 },
          { "from": 200.0 }
        ]
      }
    }
  }
}
'

The ranges in the price aggregation depend on the other hand pretty much on what is being searched. The price ranges that may be relevant to the users when searching for shoes will very likely be different from the ones that will be shown when searching for laptop because most of the shoes won’t cost more than 150 EUR, where most of the decent laptops start at 300-400 EUR.

There is currently quite an old open feature request to support dynamically calculated price ranges:

https://github.com/elastic/elasticsearch/issues/9572

This blog post tries to provide a few possible answers on how to come up with balanced ranges for the price aggregations on Elasticsearch searches.

Static price ranges

As described above, Elasticsearch already offers the ability of specifying ranges for the price in a search request. The users like to choose from a list of up to 5 price ranges in order to avoid being overwhelmed with options.

The main drawback when using this approach is that at the request time there is no way to know how the prices of the products matching the search will be distributed.

As a consequence, the price ranges used will sometimes not contain relevant information.

Use two queries to get balanced price range aggregations

Obtain first the distribution of prices for the search by using the percentiles aggregation and subsequently use this information for specifying the price ranges of the price aggregation in the second search request.

The few buckets that will be shown to the user for the price ranges will mirror the actual price distribution for the products that are matching the search.

The principal drawback in this approach is that there are two search requests needed to be made against Elasticsearch in order to obtain this information. Having two requests may very likely be considered a drawback in adopting this strategy for websites with a high amount of users performing searches.

For adding a bit of clarity here, a pseudocode version of the searches is exposed below, in the case when there are up to 3 price ranges to be shown to the user:

Search 1: Get p33, p66 percentiles aggregation for the prices matching the search
Search 2: Use [0 - p33], [p33-p66], [p66 - *] as ranges in the price aggregation

Collapse the buckets

This method builds on top of the static price ranges previously described. Instead of retrieving a handful of buckets for the price range aggregation, this method relies on specifying a lot of price range buckets (e.g. : 30/40/50) which are then collapsed/merged on the client side to a handful (e.g. : 3/4/5) of relevant price range buckets.

The main performance drawback in this approach is that there are much more price range buckets specified to be aggregated in the search request compared to the number price range buckets that are actually needed to be displayed to the user.

Source Code

Check out the source code of the project elastic-price-range-aggregation accompanying this blog post. This project contains code corresponding to each of the strategies presented in this blog post.

This project contains a JUnit testcase that spawns a testcontainers Elasticsearch container fills it with test data and tries through different strategies to retrieve the price range aggregations.

Run the command

./gradlew test

for executing the tests from this project.

Feedback

This blog post serves as a proof of concept for performing price aggregations. Eventual improvements to the project code or ideas regarding alternative ways to get the price aggregations are very much welcome.

Search Alerts

Mon, 20 Jul 2020 00:00:00 GMT

Send notifications to the users whenever new content matching their interests is added on the platform.

Search alerts are designed to monitor changes to the documents being searched, whereas search engines are designed for finding documents matching a specific search term.

Check out the blog post on percolator to get an idea on how the search upside down works in Elasticsearch:

A short primer #elasticsearch percolator https://t.co/TLglOY15Kf
— findinpath (@findinpath) July 14, 2020

This proof of concept handles both kind of notifications for its users:

immediate: as soon as new content of interest is made available
batched: hourly/ daily/ weekly in case there new content of interest is made available

For the sake of better understanding the purpose of this system there will be considered that this system handles the search alert functionality for a news platform.

The users of the news platform which have interest for a certain topic can register search alerts to be notified (e.g.: by email) when new articles matching their criteria are published on the platform. The search alerts can be configured by the users to send notifications immediately/ hourly/ daily/ weekly about new interesting content.

Technological stack

The tech stack of the presented system makes use of:

Apache Kafka for streaming data between the components of the system
Elasticsearch distributed, RESTful search engine providing percolator functionality
Apache Cassandra for storing information about the search alert messages already sent in order to avoid sending duplicated messages (e.g.: for a batched search alert on a specific hour/day).

Architecture

There are some subtle differences in handling the immediate and the batched search alerts.

The search alert immediate notifications should be sent as their name denotes as soon as a corresponding news is being published, whereas the batched notifications are to be sent only after their corresponding frequency time window (e.g.: hour/day/week) elapses.

Percolator

This is a rather simple component which consumes news records from the input Apache Kafka topic, percolates them and writes each of the corresponding search alerts retrieved from Elasticsearch to its corresponding topic to be consumed by:

the immediate-messenger component in case that the search alert has an immediate notification policy
the batched-messenger component in case that the search alert has a batched notification policy

The search alerts stored in Elasticsearch should store at a minimum meta information like:

the notification policy: immediate/hourly/daily/weekly
the email address of the user who registered the search alert

Immediate search alerts

Handling immediate search alerts is relatively straightforward by following the architecture concept diagram presented above.

Each time a new document (e.g. : news) is being percolated, for each of the corresponding search alerts there will be written to the immediate Apache Kafka topic an entry containing:

news details
search agent details

Using the search alert id as record key for the message sent towards the topic immediate ensures that the notifications sent for a search alert will be handled in a serial fashion (which comes in handy when checking for duplicates).

On the immediate-messenger side, the record will be simply sent to the mail (or push) notification service and will be subsequently recorded to the database to avoid sending duplicate messages to the user in case of errors/bugs on the percolator component side.

To avoid sending duplicates, in the Cassandra database backing the immediate-messenger component will be checked before sending a notification whether for the key:

search alert id (partition key)
news id (clustering key)

has already been recorded.

The body of the notification for the search alert is composed out of the incoming news details in the consumer record from immediate Apache Kafka topic.

Batched search alerts

If the immediate search alerts are sent straight away after percolating a new document (e.g.: news), the batched search alerts on the other hand need to be “parked” until their corresponding notification period (e.g.: hour/day/week) elapses.

Considering that a search alert is configured to hourly notify a user about new articles matching a specific search criteria, when an article is being published on 2020-06-19 09:21:05 on the news platform, the search alert should then notify the user at 2020-06-19 10:00:00.

The approach used by this proof of concept to solve this problem is to use corresponding topics for each kind of frequency window(hour/day/week) offered by the search alert service.

In the above mentioned case, the search alert information would be pushed to the topic hourly_1592553600000 (1592553600000 corresponds to 2020-06-19 10:00:00). In case that the search alert would have been configured to delivery notifications daily, then the search alert information would be then published to the topic daily_1592604000000 (1592604000000 corresponds to 2020-06-20 00:00:00).

The consumption of the entries from the hourly_1592553600000 topic will start at the earliest at 2020-06-19 10:00:00. The consumption of the entries from the daily_1592604000000 topic will start at the earliest at 2020-06-20 00:00:00.

The batched-messenger component is responsible for handling the batched search alerts from a specified topic that contain entries for the hourly/daily/weekly search alerts. The consumption from the Apache Kafka topic corresponding to a specific hour/day/week time window will begin only after its corresponding time window has elapsed. Once the consumption of the Apache Kafka topic reaches the end of the topic, the batched-messenger component instance can end its runtime because there will be no new content added to this topic in the future.

Once all the partitions of the Apache Kafka topic hourly_1592553600000 are fully read by the batched-messenger component (current offset is equal with the end offset on all the partitions of the topic), the batched topic can be considered obsolete (and can be eventually be deleted) and not being taken anymore into consideration for consumption.

To avoid sending duplicates, in the Cassandra database backing the batched-messenger component will be checked before sending a notification whether for the key:

search alert id (partition key)
frequency (clustering key. possible values: hour/day/week)

there has already been recorded a timestamp corresponding for the beginning of the frequency window that is the same (or higher) as the current frequency window being handled by the batched-messenger component.

Using the search alert id as record key for the messages sent from the percolator component towards the batched topics ensures that the notifications sent for a search alert will be handled in a serial fashion (which comes in handy when checking for duplicates).

The body of the notification for the search alert is composed out of the latest news retrieved by running the search query corresponding to the search alert (from the consumer record of the Apache Kafka batched topic) on Elasticsearch.

OPTIONAL In case of finding a matching batched search alert for a percolated news article, the percolator component should be responsible to “pause” the search alert until its current batching period (hour/day/week) elapses in order to avoid doing unnecessary matches against new incoming articles.

Orchestrating batched messengers

As mentioned previously, the batched search alert notifications need to be “parked” until their corresponding notification period(hour/day/week) elapses.

At the beginning of each batched time window (hour/day/week) should be therefor started one or multiple instances of the batched-messenger component for the batched Apache Kafka topics on which the consumer offset of the batched-messenger lags behind.

Depending on the amount of the batched search alert hits from the topic that need to be processed, the orchestrator could then choose how many instances of the batched-messenger component to spawn.

Code

Being a proof of concept, this blog post is accompanied by a the search-alert Github project which contains the following modules:

percolator
immediate-messenger
batched-messenger

which fit to the description from the Architecture section.

The search-alert Github project makes heavy use of the testcontainers library for spawning during the tests trowaway containers for:

Apache Kafka
Elasticsearch
Apache Cassandra

in order to test the functionality of the code in a realstic fashion.

NOTE: The current implementation of the mailing service from the messenger projects doesn’t actually send email notifications, but just logs them.

Open points

A key point that hasn’t been covered by the search-alert Github project accompanying this blog post is the orchestration of the batched messengers.

At the beginning of each batched time window (hour/day/week) there should run a program in a cron fashion to spawn batched-messenger instances for the previous time window. In case of eventual previous failures (when batched Apache Kafka topics having consumer offset lagging behind the end offset), this program should also spawn batched-messenger instances for other previous time windows.

The garbage collection for obsolete batched Apache Kafka topics (current offset is equal with the end offset on all the partitions of the topic) should also be taken care of by the above described program.

Probably one of the next posts will tackle this problem with kubernetes.

Elasticsearch percolator primer

Mon, 13 Jul 2020 00:00:00 GMT

Primer in understanding the reverse search concepts behind the Elasticsearch percolator.

In a classical search scenario, a term query is used as input and the documents from the search index matching the term will be returned as output. On the other hand, when percolating, a document will be used as input and the search queries registered in the search index which match the document will be returned as output.

Technically, the basic idea behind the percolator is that (as denoted by the documentation of the percolate query):

It can be used to match queries stored in an index. The percolate query itself contains the document that will be used as query to match with the stored queries.

The percolate query is called turning search upside down in the Elasticsearch in action book for the following reasons:

You index queries instead of documents. This registers the query in memory, so it can be quickly run later.

You send a document to Elasticsearch instead of a query. This is called percolating a document, basically indexing it into a small, in-memory index. Registered queries are run against the small index, so Elasticsearch finds out which queries match.

You get back a list of queries matching the document, instead of the other way around like a regular search.

The implementation of the percolator can be found in the percolator module from the elasticsearch Github repository.

How does the percolator actually work

Initially, when the percolator was released back in the version 0.15.0. of Elasticsearch, the execution time of the percolator was always linear to the amount of percolator queries, because all percolator queries had to be evaluated all the time.

When percolating, the document being percolated gets indexed into temporary in-memory index. Prior to version 5.0 of Elsticsearch, all percolator queries needed to be executed on an in-memory index in order to verify whether the query matches.

As of version 5.0 of Elasticsearch there have been added dramatic improvements on how the percolator works by being able to skip evaluating most queries against the in-memory index.

The percolator analyses the queries and creates appropriate data structures in order to be able to filter only the relevant search queries that need to be executed on the memory index containing the percolated document:

for a term (e.g. : search for all the documents containing elastic ) query there will be used a org.apache.lucene.search.TermScorer that will make use of the Lucene inverted index functionality for retrieving only the documents that contain the searched term
for a int field (e.g. : search all the real estate object with 4 rooms) an intersection will be made between the range of the percolated document (e.g. : 4 to 4) and the KD tree containing the ranges (e.g. : 0 TO 3 , 4 TO *, 2 TO 5, etc.)

NOTE: Some query types can’t be filtered (e.g. : wildcard queries) reason why they will be need to be executed against the memory index to check whether they match the percolated document.

The percolator classes relevant for understanding how the queries get analysed and converted into documents belonging to a query search index are:

Try out the tests from the elastic-percolator-primer Java project in Debug mode with breakpoints in the previously mentioned percolator classes to get a hands-on understanding how the query search indexes used in the percolation process are built and used.

Usage demo

This demo is based on the blog post

https://sharing.luminis.eu/blog/using-the-new-elasticsearch-5-percolator/

and contains just simple modifications in order to get the queries compatible with Elasticsearch 7.x.

Use either the standalone installation or the docker image for Elasticsearch to get started.

Create `news` index

# Create news index and populate it with documents

curl -X PUT "http://localhost:9200/news?pretty" -H 'Content-Type: application/json' -d'
{
  "mappings": {
    "properties": {
      "title": {"type": "text"},
      "body": {"type": "text"},
      "category": {"type": "keyword"}
    }
  }
}'

Populate `news` index with documents

curl -X PUT "http://localhost:9200/news/_doc/1" -H 'Content-Type: application/json' -d'
{
  "title": "Early snow this year",
  "body": "After a year with hardly any snow, this is going to be a serious winter",
  "category": "weather"
}'


curl -X PUT "http://localhost:9200/news/_doc/2" -H 'Content-Type: application/json' -d'
{
  "title": "Snow on the ground, sun in the sky",
  "body": "I am waiting for the day where kids can skate on the water and the dog can play in the snow while we are sitting in the sun.",
  "category": "weather"
}'


curl -X PUT "http://localhost:9200/news/_doc/3" -H 'Content-Type: application/json' -d'
{
  "title": "Snow everywhere",
  "body": "Everything is covered in snow. The sun is shining.",
  "category": "weather"
}'

Basic search operations on the `news` index

# Get one document by id

curl -X GET "localhost:9200/news/_doc/1?pretty"

# Retrieve all documents

curl -X GET "localhost:9200/news/_search?pretty" -H 'Content-Type: application/json' -d'
{
  "query": { "match_all": {} }
}
'

Create the `news-notify` index

As recommended in the documentation of search_after there will be used a special field tie_breaker_id for the search queries to be employed in ordering the paged percolation results.

curl -X PUT "localhost:9200/news-notify?pretty" -H 'Content-Type: application/json' -d'
{
    "mappings": {
        "properties": {
             "title": {
                 "type": "text"
             },
             "category": {
                 "type": "keyword"
             },
             "tie_breaker_id": {
                "type": "long"
             },
             "query": {
                 "type": "percolator"
             }
        }
    }
}
'

Populate `news-notify` index

curl -XPUT "http://localhost:9200/news-notify/_doc/1" -H 'Content-Type: application/json' -d'
{
  "query": {
    "bool": {
      "must": [
        {
          "match": {
            "title": "snow"
          }
        }
      ],
      "filter": [
        {
          "term": {"category":  "weather"}
        }
      ]
    }
  },
  "tie_breaker_id": 1,
  "meta": {
    "username": "sander",
    "create_date": "2016-10-13T14:23:00"
  }
}'

curl -XPUT "http://localhost:9200/news-notify/_doc/2" -H 'Content-Type: application/json' -d'
{
  "query": {
    "bool": {
      "must": [
        {
          "match": {
            "title": "sun"
          }
        }
      ],
      "filter": [
        {
          "term": {
            "category": {
              "value": "weather"
            }
          }
        }
      ]
    }
  },
  "tie_breaker_id": 2,
  "meta": {
    "username": "jettro",
    "create_date": "2016-10-13T14:21:45"
  }
}'

curl -XPUT "http://localhost:9200/news-notify/_doc/3" -H 'Content-Type: application/json' -d'
{
  "query": {
    "bool": {
      "must": [
        {
          "match": {
            "title": "sun"
          }
        }
      ],
      "filter": [
        {
          "term": {
            "category": {
              "value": "weather"
            }
          }
        }
      ]
    }
  },
  "tie_breaker_id": 3,
  "meta": {
    "username": "lara",
    "create_date": "2016-10-13T14:21:45"
  }
}'

curl -XPUT "http://localhost:9200/news-notify/_doc/4" -H 'Content-Type: application/json' -d'
{
  "query": {
    "bool": {
      "must": [
        {
          "match": {
            "title": "snow"
          }
        }
      ],
      "filter": [
        {
          "term": {
            "category": {
              "value": "weather"
            }
          }
        }
      ]
    }
  },
  "tie_breaker_id": 4,
  "meta": {
    "username": "simon",
    "create_date": "2016-10-13T14:21:45"
  }
}'

Percolate queries

Specify the document to be percolated within the query:

curl -XGET "http://localhost:9200/news-notify/_search" -H 'Content-Type: application/json' -d'{
   "query":{
      "percolate":{
         "field":"query",
         "document":{
           "title" : "Early snow this year",
           "body" : "After a year with hardly any snow, this is going to be a serious winter",
           "category" : "weather"
         }
      }
   }
}'

or reference it from the news index (if it has already been indexed):

curl -XGET "http://localhost:9200/news-notify/_search" -H 'Content-Type: application/json' -d'{
   "query":{
      "percolate":{
         "field":"query",
         "index":"news",
         "id":"1"
      }
   }
}'

Paging of the percolation results with `search_after`

Pagination of results can be done by using the from and size, but the cost becomes prohibitive when the deep pagination is reached. Take a look at the search_after documentation to see how this problem can be circumvented.

curl -XGET "http://localhost:9200/news-notify/_search" -H 'Content-Type: application/json' -d'
{
  "query": {
    "percolate": {
      "field": "query",
      "index": "news",
      "id": "2"
    }
  },
  "sort": [
        {"tie_breaker_id": "asc"}
  ],
  "size": 1
}'


# search_after will be populated in the with the ID of the last
# document retrieved in the last query (e.g. : `1`)


curl -XGET "http://localhost:9200/news-notify/_search" -H 'Content-Type: application/json' -d'
{
  "query": {
    "percolate": {
      "field": "query",
      "index": "news",
      "id": "2"
    }
  },
  "sort": [
        {"tie_breaker_id": "asc"}
  ],
  "search_after":[1],
  "size": 1
}'

Delete indexes

curl -X DELETE "localhost:9200/news?pretty"

curl -X DELETE "localhost:9200/news-notify?pretty"

Resources

Announcement of the release of the Elasticsearch percolator

https://www.elastic.co/blog/percolator

When and how to percolate

Introduction of the percolate query

https://www.elastic.co/blog/elasticsearch-percolator-continues-to-evolve

Code

Try out the tests from the Github project elastic-percolator-primer to get a feeling about how the percolator works.

JDBC Nested Set Sink Connector

Sun, 24 May 2020 00:00:00 GMT

Demo showcase of JDBC Nested Set Sink Connector for Apache Kafka Connect used to eventually sync hierachical data (e.g. : shop category tree) from Apache Kafka towards a sink database table via Apache Kafka Connect in an untainted fashion without intermittently having corrupt content on the sink database destination nested set model table.

Nested Set Model

There are multiple ways of storing and reading hierarchies in a relational database:

adjacency list model: each tuple has a parent id pointing to its parent
nested set model: each tuple has left and right coordinates corresponding to the preordered representation of the tree

Details about the advantages of the nested set model are already very well described in the following article:

https://www.sitepoint.com/hierarchical-data-database/

TLDR As mentioned on Wikipedia

The nested set model is a technique for representing nested sets (also known as trees or hierarchies) in relational databases.

Syncing nested set models over Apache Kafka

Kafka Connect is an open source component of Apache Kafka which in a nutshell, as described on Confluent blog provides the following functionality for databases:

It enables you to pull data (source) from a database into Kafka, and to push data (sink) from a Kafka topic to a database.

The topic of synchronizing nested set model data between databases has been already discussed in the blog post :

Ever wondered how to sync trees or hierarchies over @apachekafka by making use of @confluentinc kafka-connect-jdbc ? https://t.co/p85iH3a6Yu
— findinpath (@findinpath) April 19, 2020

In the previous blog post the nested set model data published over kafka-connect-jdbc source connector from the source database to Apache Kafka was being handled explicitly in the business logic of the sink microservice application.

It is definitely possible to integrate the synchronization logic for the nested set data explicitly in a microservice that depends on the nested set model data (e.g. : shop category tree), but this adds, possibly unwanted, further complexity to the microservice.

The synchronization functionality for the nested set model data which is a non-core functionality of the microservice needs to be maintained and monitored.

Another possibility would be to delegate the responsibility of syncing the nested set model data to a JDBC Sink Connector from the Confluent Hub. This approach would have the advantage that the microservice consuming the nested set model data would solely concentrate on its core functionality.

This post describes how to use the JDBC Nested Set Sink Connector created to generically sink nested set model data via Apache Kafka Connect in a destination nested set model table.

End to end synchronization over Apache Kafka Connect

Syncing of nested set model from the source database to Apache Kafka can be easily taken care of by a kafka-connect-jdbc source connector which can be initialized by posting the following configuration to the /connectors endpoint of Kafka Connect (see Kafka Connect REST interface)

{
    "name": "findinpath",
    "config": {
        "connector.class": "io.confluent.connect.jdbc.JdbcSourceConnector",
        "mode": "timestamp+incrementing",
        "timestamp.column.name": "updated",
        "incrementing.column.name": "id",
        "topic.prefix": "findinpath.",
        "connection.user": "sa",
        "connection.password": "p@ssw0rd!source",
        "validate.non.null": "false",
        "tasks.max": "1",
        "name": "findinpath",
        "connection.url": "jdbc:postgresql://source:5432/source?loggerLevel=OFF",
        "table.whitelist": "nested_set_node"
    }
}

NOTE in the configuration above, the tasks.max is set to 1 because JDBC source connectors can deal only with one SELECT statement at a time for retrieving the updates performed on a table. It is advisable to use also for Apache Kafka a topic with only 1 partition for syncing the nested set content towards downstream services.

Syncing of the nested set model data from Apache Kafka towards the sink database can be now easily taken care of by the JDBC Nested Set Sink Connector:

{
    "name": "jdbc-nested-set-node-sink",
    "config": {
        "name": "jdbc-nested-set-node-sink",
        "connector.class": "com.findinpath.connect.nestedset.jdbc.NestedSetJdbcSinkConnector",
        "tasks.max": "1",
        "topics": "findinpath.nested_set_node",
        "connection.url": "jdbc:postgresql://sink:5432/postgres",
        "connection.user": "sa",
        "connection.password": "p@ssw0rd!sink",
        "pk.fields": "id",
        "table.name": "nested_set_node",
        "table.left.column.name": "lft",
        "table.rgt.column.name": "rgt",
        "log.table.name": "nested_set_node_log",
        "log.table.primary.key.column.name": "log_id",
        "log.table.operation.type.column.name": "operation_type",
        "log.offset.table.name": "nested_set_node_log_offset",
        "log.offset.table.log.table.column.name": "log_table_name",
        "log.offset.table.offset.column.name": "log_table_offset"
    }
}

Functionality overview

The JDBC Nested Set Sink Connector is relatively similar in functionality to the JDBC Sink Connector for Confluent Platform because it ingests SinkRecord entries and writes them in a database table. This is also the reason why this connector made use of a great part of the sink logic of the kafka-connect-jdbc project code.

The JDBC Nested Set Sink Connector writes the sink records in a database table structure similar to the one shown below:

The nested_set_node_log table is an INSERT only table which simulates a certain extent the transaction logs on the nested set model data.

After writing new entries on the nested_set_node_log table, there will be an attempt to synchronize them towards the destination nested_set_node table. This operation will succeed only when the nested set model resulting from the merge of:

existing content from the nested_set_node table
new nested_set_node_log table entries

is valid.

The table nested_set_node_log_offset will contain only a pointer towards the ID of the latest nested_set_node_log entry synchronized successfully into the nested_set_node table.

NOTE: The microservice that makes use of the nested set model tree content should continuously poll the table nested_set_node_log_offset in order to know when the nested set model has been updated.

At the time of this writing the connector supports and has been tested with the following databases:

Postgres 12
MySQL 8
MS SQL Server 2017
sqlite 3
Oracle 18.4.0 XE

The JDBC Nested Set Sink Connector supports both:

upsert: the position of the node in the tree or its data can be upserted (updated/inserted)
deletion: the node can be removed from the tree (works with Debezium change data capture Kafka Connect source connector)

operations on the nested set model entries.

Testing

It is relatively easy to think about a solution for the previously exposed problem, but before putting it to a production environment the solution needs proper testing in conditions similar to the environment in which it will run.

This is where the testcontainers library helps a great deal by providing lightweight, throwaway instances of common databases that can run in a Docker container.

Docker containers are used for interacting with the Apache Kafka ecosystem as well as the source and sink databases.

This leads to tests that are easy to read and allow the testing of the sync operation for various nested set models

    /**
     *   Ensure that the sync the content of a more complex nested set model
     *   is performed successively each time after performing updates on the
     *   nested set model on the source database.
     */
    @Test
    public void syncingSuccessiveChangesToTheTreeDemo() {

        long clothingNodeId = sourceNestedSetService.insertRootNode("Clothing");
        awaitForTheSyncOfTheNode(clothingNodeId);
        logSinkTreeContent();

        // The current structure of the tree should be:
        // |1| Clothing |2|


        // Add now Men's and Women's children and wait for the syncing
        long mensNodeId = sourceNestedSetService.insertNode("Men's", clothingNodeId);
        long womensNodeId = sourceNestedSetService.insertNode("Women's", clothingNodeId);

        awaitForTheSyncOfTheNode(womensNodeId);
        logSinkTreeContent();

        // The current structure of the tree should be:
        // |1| Clothing |6|
        //     ├── |2| Men's |3|
        //     └── |4| Women's |5|



        // Add new children categories for both Men's and Women's nodes
        sourceNestedSetService.insertNode("Suits", mensNodeId);
        sourceNestedSetService.insertNode("Dresses", womensNodeId);
        sourceNestedSetService.insertNode("Skirts", womensNodeId);
        sourceNestedSetService.insertNode("Blouses", womensNodeId);

        awaitForTheSyncOfTheNode(womensNodeId);
        logSinkTreeContent();
        // The current structure of the tree should be:
        //   |1| Clothing |14|
        //       ├── |2| Men's |5|
        //       │       └── |3| Suits |4|
        //       └── |6| Women's |13|
        //               ├── |7| Dresses |8|
        //               ├── |9| Skirts |10|
        //               └── |11| Blouses |12|
    }

See DemoNestedSetSyncTest for several syncing test cases.

This project provides a functional prototype on how to setup the whole Confluent environment (including Confluent Schema Registry and Apache Kafka Connect) via testcontainers.

See AbstractNestedSetSyncTest for details.

Source code

Checkout the github project kafka-connect-nested-set-jdbc-sink.

Try out the tests via

mvn clean test

Installation notes

The connector is available on Confluent Hub and can be installed via the following command:

confluent-hub install findinpath/kafka-connect-nested-set-jdbc-sink:1.0.0

Sync tree database table over Apache Kafka

Sun, 19 Apr 2020 00:00:00 GMT

Showcase on how to eventually sync hierachical data from a source database table towards a sink database table via Apache Kafka in a untainted fashion without intermittently having corrupt content on the sink database table.

UPDATE 2020-05-28

The problem showcased here can be solved also by using the new JDBC Nested Set Sink Connector for Kafka Connect. Check out the following blog post for more details on the JDBC Nested Set Sink Connector :

Showcase of JDBC Nested Set Sink Connector for @confluentinc platform. https://t.co/rGxIk3HPRy
— findinpath (@findinpath) May 26, 2020

Nested Set Model

There are multiple ways of storing and reading hierarchies in a relational database:

adjacency list model: each tuple has a parent id pointing to its parent
nested set model: each tuple has left and right coordinates corresponding to the preordered representation of the tree

Details about the advantages of the nested set model are already very well described in the following article:

https://www.sitepoint.com/hierarchical-data-database/

TLDR As mentioned on Wikipedia

The nested set model is a technique for representing nested sets (also known as trees or hierarchies) in relational databases.

Syncing nested set models over Apache Kafka

Kafka Connect is an open source component of Apache Kafka which in a nutshell, as described on Confluent blog provides the following functionality for databases:

It enables you to pull data (source) from a database into Kafka, and to push data (sink) from a Kafka topic to a database.

More details about kafka-connect-jdbc connector can be found on the Conflent documentation.

Syncing of nested set model from the source database to Apache Kafka can be easily taken care of by a kafka-connect-jdbc source connector which can be initialized by posting the following configuration to the /connectors endpoint of Kafka Connect (see Kafka Connect REST interface)

{
    "name": "findinpath",
    "config": {
        "connector.class": "io.confluent.connect.jdbc.JdbcSourceConnector",
        "mode": "timestamp+incrementing",
        "timestamp.column.name": "updated",
        "incrementing.column.name": "id",
        "topic.prefix": "findinpath.",
        "connection.user": "sa",
        "connection.password": "p@ssw0rd!source",
        "validate.non.null": "false",
        "tasks.max": "1",
        "name": "findinpath",
        "connection.url": "jdbc:postgresql://source:5432/source?loggerLevel=OFF",
        "table.whitelist": "nested_set_node"
    }
}

Refresh the nested set model on the sink database

On the sink database side there needs to be implemented a mechanism that contains a safeguard for not adding invalid updates to the nested set model. A concrete example in this direction is when going from the nested set model:

|1| A |2|

to the nested set model (after adding two children)

|1| A |6|
    ├── |2| B |3|
    └── |4| C |5|

In the snippets above the tree node labels are prefixed by their left and right nested set model coordinates.

Via kafka-connect-jdbc the records corresponding to the tuple updates may come in various ordering:

| label | left | right |
|-------|------|-------|
| A     | 1    | 6     |
| B     | 2    | 3     |
| C     | 4    | 5     |

| label | left | right |
|-------|------|-------|
| B     | 2    | 3     |
| C     | 4    | 5     |
| A     | 1    | 6     |

or any other combinations of the three tuples listed above because the records are polled in batches of different sizes from Apache Kafka.

Going from the nested set model table content

| label | left | right |
|-------|------|-------|
| A     | 1    | 2     |

towards

| label | left | right |
|-------|------|-------|
| A     | 1    | 6     |

| label | left | right |
|-------|------|-------|
| A     | 1    | 6     |
| B     | 2    | 3     |

would intermittently render the nested set model corrupt until all the records from the source nested set model are synced over Apache Kafka.

Using a kafka-connect-jdbc sink connector is therefor out of the question for syncing the contents of trees from a source service towards downstream online services.

One solution to cope with such a problem would be to separate the nested set model from what is being synced over Apache Kafka.

In the table diagram above, the nested_set_node_log table is an INSERT only table in which is written whenever a new record(s) is read from Apache Kafka. The log_offset table has only one tuple pointing to the last nested_set_node_log tuple id processed when updating the nested_set_node table.

Whenever new records are read from Apache Kafka, there will be a transactional attempt to apply all the updates from nested_set_node_log made after the saved entry in the log_offset table to the existing configuration of the nested_set_node nested set model.

If the applied updates lead to a valid nested set model configuration, then the nested_set_node table will be updated and the log offset will be set to the latest processed nested_set_node_log entry, otherwise the nested_set_node table stays in its previous state.

Caching

On the sink side is implemented the Guava’s Cache for avoiding to read each time from the persistence the contents of the nested set model. This approach nears the usage on a productive system, where the contents of the nested set model are cached and not read from the database for each usage.

When there are new contents added to the nested set model, the cache is notified for invalidating its contents.

JDBC Transactions

One of the challenges faced before implementing this proof of concept was whether to use spring framework to wrap the complexities of dealing with JDBC. It is extremely appealing to use production ready frameworks and not care about their implementation complexity.

The decision made in the implementation was to avoid using spring and JPA and go with plain old JDBC.

Along the way in the implementation, one open question was whether to group all the JDBC complexity in one repository or in multiple repositories. Due to the fact that multiple repositories bring a better overview in the maintenance, the decision was made to go with multiple repositories.

There were some scenarios which involved transaction handling over multiple DAO objects. The possible ways of handling transactions over multiple repositories is very well described in the stackexchange post:

https://softwareengineering.stackexchange.com/a/339458/363485

The solution used to cope with this situation within this proof of concept was to create repositories for each service operation and inject the connection in the repositories.

Dependency injection of connection: Your DAOs are not singletons but throw-away objects, receiving the connection on creation time. The calling code will control the connection creation for you.

PRO: easy to implement

CON: DB connection preparation / error handling in the business layer

CON: DAOs are not singletons and you produce a lot of trash on the heap (your implementation language may vary here)

CON: Will not allow stacking of service methods

Testing

It is relatively easy to think about a solution for the previously exposed problem, but before putting it to a production environment the solution needs propper testing in conditions similar to the environment in which it will run.

This is where the testcontainers library helps a great deal by providing lightweight, throwaway instances of common databases that can run in a Docker container.

More details on how to perform integration tests involving kafka-connect-jdbc can be found on the blog post:

Showcase of testing syncing of #postgres
data via @confluentinc kafka-connect-jdbc towards @apachekafka with @testcontainers https://t.co/HdwH12neDW
— findinpath (@findinpath) April 10, 2020

Docker containers are used for interacting with the Apache Kafka ecosystem as well as the source and sink databases.

This leads to tests that are easy to read and allow the testing of the sync operation for various nested set models

    /**
     *   Ensure that the sync the content of a more complex nested set model
     *   is performed successively each time after performing updates on the
     *   nested set model on the source database.
     */
    @Test
    public void syncingSuccessiveChangesToTheTreeDemo() {

        var clothingNodeId = sourceNestedSetService.insertRootNode("Clothing");
        awaitForTheSyncOfTheNode(clothingNodeId);
        logSinkTreeContent();

        // The current structure of the tree should be:
        // |1| Clothing |2|


        // Add now Men's and Women's children and wait for the syncing
        var mensNodeId = sourceNestedSetService.insertNode("Men's", clothingNodeId);
        var womensNodeId = sourceNestedSetService.insertNode("Women's", clothingNodeId);

        awaitForTheSyncOfTheNode(womensNodeId);
        logSinkTreeContent();

        // The current structure of the tree should be:
        // |1| Clothing |6|
        //     ├── |2| Men's |3|
        //     └── |4| Women's |5|



        // Add new children categories for both Men's and Women's nodes
        sourceNestedSetService.insertNode("Suits", mensNodeId);
        sourceNestedSetService.insertNode("Dresses", womensNodeId);
        sourceNestedSetService.insertNode("Skirts", womensNodeId);
        sourceNestedSetService.insertNode("Blouses", womensNodeId);

        awaitForTheSyncOfTheNode(womensNodeId);
        logSinkTreeContent();
        // The current structure of the tree should be:
        //   |1| Clothing |14|
        //       ├── |2| Men's |5|
        //       │       └── |3| Suits |4|
        //       └── |6| Women's |13|
        //               ├── |7| Dresses |8|
        //               ├── |9| Skirts |10|
        //               └── |11| Blouses |12|
    }

See DemoNestedSetSyncTest for several syncing test cases.

This project provides a functional prototype on how to setup the whole Confluent environment (including Confluent Schema Registry and Apache Kafka Connect) via testcontainers.

See AbstractNestedSetSyncTest and the testcontainers package for details.

Kafka Connect

In order to use the Confluent’s Kafka Connect container, this project made use of the already existing code for KafkaConnectContainer on ydespreaux Github account.

NOTE that the KafkaConnectContainer class previously mentioned has also corresponding test cases within the project lib-kafka-connect in order to have a clue on how to interact in an integration test with the container.

Sample code

Checkout the github project sample project nested-set-kafka-sync and try out the tests via

mvn clean test

kafka-connect-jdbc testcontainers tests

Fri, 10 Apr 2020 00:00:00 GMT

Test a system that depends on kafka-connect-jdbc by bootstrapping the Apache Kafka ecosystem artifacts and PostgreSQL through testcontainers.

In a nutshell, as described on Confluent blog the kafka-connect-jdbc provides the following functionality:

It enables you to pull data (source) from a database into Kafka, and to push data (sink) from a Kafka topic to a database.

This post is a showcase on how to test the synchronization of the contents of a PostgreSQL table via kafka-connect-jdbc towards Apache Kafka.

The contents of the input PostgreSQL table are synced as AVRO messages towards Apache Kafka.

The showcased project testcontainers-kafka-connect makes use of docker containers (via testcontainers library) for showcasing the Confluent Kakfa Connect functionality in an automated test case.

The interactions from this proof of concept are described visually in the image below:

The proof of concept testcontainers-kafka-connect project can be used for making end to end system test cases with docker for architectures that rely on kafka-connect-jdbc for syncing content from a relational database (PostgreSQL is the database used in the aforementioned project).

The testcontainers library already offers a Kafka module for interacting with Apache Kafka, but there is not, at the moment, a testcontainers module for the whole Confluent environment (Confluent Schema Registry / Apache Kafka Connect container support is missing from the module previously mentioned).

The project testcontainers-kafka-connect contains custom implementations for testcontainers extensions corresponding for:

Apache Zookeeper
Apache Kafka
Confluent Schema Registry
Apache Kafka Connect

that can be used to reproduce a multi-container scenario that involves working with the aforementioned components of the Apache Kafka ecosystem.

As a side note, the containers used do not use the default ports exposed by default in the artifacts (e.g. : Apache Zookeeper 2181, Apache Kafka 9092, Confluent Schema Registry 8081, Apache Kafka Connect 8083), but rather free ports available on the test machine avoiding therefor possible conflicts with already running services on the test machine.

For the test environment the following containers will be started:

Apache Zookeeper
Apache Kafka
Confluent Schema Registry
Confluent Kafka Connect
PostgreSQL

It is quite impressive to see how close a productive environment can be simulated in the test cases with the testcontainers library.

Demo test

Once the test environment is started, via a HTTP call performed with rest-assured a kafka-connect-jdbc connector will be registered for the bookmarks PostgreSQL table.

The kafka-connect-jdbc connector will afterwards then continously poll the bookmarks table and will sync its contents towards the findinpath.bookmarks Apache Kafka topic.

The demo verifies whether the dynamically inserted contents into the bookmarks Postgres table get successfully synced in AVRO format on the Apache Kafka topic findinpath.bookmarks in the same order as they were inserted.

Sample code

Checkout the github project sample project testcontainers-kafka-connect and try out the tests via

mvn clean test

Dynamic property sources for PostgreSQL spring boot tests

Sun, 29 Mar 2020 00:00:00 GMT

Make use of the newly introduced DynamicPropertySource spring annotation in configuring DataSource spring beans required in the tests that make use of testcontainers.

Spring Boot interacts with databases via with data sources mapped via the spring.datasource in the application.yml file. In the Spring Boot internals, via org.springframework.boot.autoconfigure.jdbc.DataSourceInitializationConfiguration, is triggered the creation of the much needed spring bean instance of type javax.sql.DataSource for JDBC/JPA interactions.

In the context of testing DAO (Data Access Object) classes, it is very common to make use of the testcontainers library.

Testcontainers is a Java library that supports JUnit tests, providing lightweight, throwaway instances of common databases

Before introducing the annotation @DynamicPropertySource there was needed an implementation of the ApplicationContextInitializer to introduce dynamicaly the required properties in the configurable application context for bootstrapping the creation of the data source backed (via testcontainers library) by a database living in a docker container :

@SpringJUnitConfig(classes = {Application.class}, initializers = {PostgresIntegrationTest.Initializer.class})
@Testcontainers
public class PostgresIntegrationTest {

  @Container
  public static PostgreSQLContainer postgreSQLContainer = new PostgreSQLContainer("postgres:12")
      .withDatabaseName("integration-tests-db")
      .withUsername("sa")
      .withPassword("sa");


  @Autowired
  private JdbcTemplate jdbcTemplate;

  static class Initializer implements ApplicationContextInitializer<ConfigurableApplicationContext> {
    public void initialize(ConfigurableApplicationContext configurableApplicationContext) {
      TestPropertyValues.of(
          "spring.datasource.url=" + postgreSQLContainer.getJdbcUrl(),
          "spring.datasource.username=" + postgreSQLContainer.getUsername(),
          "spring.datasource.password=" + postgreSQLContainer.getPassword()
      ).applyTo(configurableApplicationContext.getEnvironment());
    }
  }

  @Test
  public void demo(){
     jdbcTemplate.execute("SELECT 1");
  }
}

With the introduction of the @DynamicPropertySource there is no need for an extra ApplicationContextInitializer:

@SpringJUnitConfig(classes = {Application.class})
@Testcontainers
public class PostgresIntegrationTest {

  @Container
  public static PostgreSQLContainer postgreSQLContainer = new PostgreSQLContainer("postgres:12")
      .withDatabaseName("integration-tests-db")
      .withUsername("sa")
      .withPassword("sa");


  @Autowired
  private JdbcTemplate jdbcTemplate;

  @DynamicPropertySource
  static void postgresProperties(DynamicPropertyRegistry registry) {
    registry.add("spring.datasource.url", postgreSQLContainer::getJdbcUrl);
    registry.add("spring.datasource.username", postgreSQLContainer::getUsername);
    registry.add("spring.datasource.password", postgreSQLContainer::getPassword);
  }

  @Test
  public void demo() {
    jdbcTemplate.execute("SELECT 1");
  }
}

As can be seen from above, the newly introduced @DynamicPropertySource is somehow similar to the commonly used @TestPropertySource annotation with the mention that it allows the usage of dynamic resources such as the IP and port assigned to the container (needed in the jdbcUrl in the example above).

Check out the full source code (and corresponding documentation) of the PostgresIntegrationTest.java class.

See more details about the usage of the @DynamicPropertyResource in the Spring framework documentation.

Sample code

Checkout the github project sample project postgres-spring-boot-dynamicpropertysource and try out the tests via

mvn clean test

Process generic avro records in Kafka Streams

Sat, 28 Dec 2019 00:00:00 GMT

Via Kafka Streams can be processed avro records of different types in order to benefit of the ordering of the events that relate to the same domain entity.

As described in Martin Kleppmann’s article Should you put several event types in the same Kafka topic there are good reasons why it would make sense to stuff multiple event types in the same Kafka topic:

The most important rule is that any events that need to stay in a fixed order must go in the same topic (and they must also use the same partitioning key). Most commonly, the order of events matters if they are about the same entity.

If the data encoding for the records passed through Apache Kafka topic is JSON it is then relatively easy to setup an application consuming the records. The problems occur when there are modifications brought to the schema of the records in order to match the new business requirements of the system because these changes make it difficult to know what version of the JSON objects are we dealing with Different clients can very likely use different versions for the schema of the JSON object that they send for processing, but the Kafka application on the receiving end needs to be able to accurately process ALL the JSON objects. This is why the Apache Kafka documentation highly encourages the usage of the avro library for data encoding of the records passed through Apache Kafka topics.

Processing generic records comes at the cost of having a higher complexity on the Apache Kafka consumer side, but having the ability to keep in order the events happening on a domain entity is worth the trouble. A concrete example to showcase the need to keep in order the events for an entity would be the registration of a new user of a site and subsequently the change of the address of the user via his profile page. In this case, the Apache Kafka application consuming the events related to users may process an address change for a user that does not exist if the event corresponding for the creation of the new user has been delayed.

This article is accompanied by the kafka-streams-generic-avro sample project which showcases the strategies available for Kafka Streams to cope with generic avro records.

The topologies showcased in the project are overly simplistic with the sole purpose of echoing the information that they receive for processing to the destination topics.

    records
        .peek((key, record) -> LOGGER
            .info("Processing entry with the key " + key + " and value " + record))
        .to(outputTopic);

The test code based on testcontainers corresponding to these topologies is used to showcase the differences between the two strategies available for processing generic avro records:

GenericRecord
SpecificRecord

Generic Record Topologies

The main advantages of building topologies based on the type org.apache.avro.generic.GenericRecord is that there can be processed virtually any kind of messages by the client. This is pretty much similar to processing JSON objects without prior knowledge of their types.

Specific Record Topologies

When consuming specific records, based on the type org.apache.avro.generic.SpecificRecord, the code of the topology has the benefit of working with typed records, which can ease up the handling of these records. The one possible inconvenient in this case is that the topology must have in its classpath the types of the records that it intends to process.

Pattern for handling the records

A possibility to handle a finite amount of record types is to have a series of if statements chained together:

   if (BookmarkEvent.equals(record.getClass()){
       // ...
   } else if (UserCreatedEvent.equals(record.getClass())){
       // ...
   } else{

   }

The main problem with this approach is that the more record types the topology tends to handle, the longer (and error prone) this handling code based on the type gets.

An alternative to chained if statements is the usage of a handler map:

    final Map<Class<? extends SpecificRecord>, BiConsumer<String, SpecificRecord>> handlers = new HashMap<>();

    handlers.put(BookmarkEvent.class, (key, record)-> LOGGER
        .info("Processing bookmark entry with the key " + key + " and value " + record));

    handlers.put(UserCreatedEvent.class, (key, record)-> LOGGER
        .info("Processing user created entry with the key " + key + " and value " + record));

    records
        .peek((key, record) -> handlers
            .getOrDefault(record.getClass(),
                (k, r) -> LOGGER.error("Handler not configured for the record " + record + " with key "+ key + " of type " + record.getClass()))
            .accept(key, record))
        .to(outputTopic);

Schema Registry settings

As already mentioned in the article Should you put several event types in the same Kafka topic in order to deal with several event types in the same topic there are two options for naming the avro subjects in the Confluent Schema Registry:

io.confluent.kafka.serializers.subject.RecordNameStrategy
io.confluent.kafka.serializers.subject.TopicRecordNameStrategy

In the sample project kafka-streams-generic-avro the RecordNameStrategy was used for naming the subjects corresponding to the record values that are written to the input Apache Kafka topic.

    streamsConfiguration.put(AbstractKafkaAvroSerDeConfig.VALUE_SUBJECT_NAME_STRATEGY,
        RecordNameStrategy.class);

Sample code

Checkout the github project sample project kafka-streams-generic-avro and try out the GenericKafkaStreamsAvroDemoTest test case to get see the concepts exposed above in action.

Bind spring-boot configuration properties in Kotlin

Thu, 28 Nov 2019 00:00:00 GMT

This post showcases how to bind @ConfigurationProperties in spring-boot projects written in kotlin.

Spring Boot offers the @ConfigurationProperties annotation in order to allow binding and validating properties set externally (e.g. via application.yml file).

When writing a Spring Boot application in Kotlin programming language, the usage of data classes comes like a good candidate to be used for binding external properties that are to be used subseqently in the business logic code of the spring beans.

Nevertheless, using an approach like the following

data class Project(val name: String, val code: String)

will result at the start of the spring boot application in the following binding error:

The elements [project.code,project.name] were left unbound.

This happens because the binding is done by property setters (which can’t succeed when using val fields in kotlin).

There are several solutions to solve the binding of the external properties in kotlin classes.

The outcome of solutions is that the fields of the Project class will be kotlin friendly non-null fields and that the bootstrap of the spring application will fail in case that the required properties are not specified.

ConstructorBinding

By using the @ConstructorBinding annotation in the data class we can indicate that configuration properties should be bound using constructor arguments rather than by calling setters.

import org.springframework.boot.context.properties.ConstructorBinding

@ConstructorBinding
data class Project(val name: String, val code: String)

Lateinit properties

Use lateinit modifier for allowing the properties of the class to be set at a later time.

class Project{
    lateinit var name: String
    lateinit var code: String


    override fun toString(): String {
        return "Project(name='$name', code='$code')"
    }
}

Testing spring-retry functionality with Cucumber

Sat, 23 Nov 2019 00:00:00 GMT

This post is intended to be heads up on the benefits in the readability of the test scenarios that come when using Cucumber library.

The Github project spring-retry-cucumber which accompanies this blog post represents a simplisting Github API client that exposes operations for retrieving Github user details.

Feel free to checkout the source code of the project and run the tests by using the command:

mvn clean test

Below is presented a simple demo of the functionality of this Github API client.

    var config = new GithubProperties("https://api.github.com/");
    var github = new Github(config);
    var usersEndpoint = github.users();

    System.out.println(usersEndpoint.getUser("findinpath"));
    System.out.println(usersEndpoint.getUsers(0));

In case that sporadic exceptions may occur on the Github API, through the usage of the spring-retry mechanism built on top of the previously demoed Github API client,these failures should go unnoticed in the client program flow.

The program flow will retrieve successfully the user details even though sometimes one sporadic API call will fail, because the API call will be retried and therefor in the client context, the API call will appear as successful (even though it was actually performed twice).

Cucumber

Cucumber is a software library that enables the usage of Behaviour Driven Development in the projects where it is integrated.

Gherkin is a core component for Cucumber responsible for parsing the executable test specifications is . Gherkin is a set of grammar rules that makes plain text structured enough for Cucumber to understand.

Below can be seen the accuracy & failure test scenarios written for the spring-retry-cucumber demo project.

When having a quick look over the scenarios above, it gets strikingly clear how the Github API client is expected to work.

The demo project spring-retry-cucumber has a few particularities in terms of usage for the Cucumber framework which will be detailed in the lines below.

Support for custom parameter types

In the images above that showcase the accuracy & failures test scenarios for the Github API client, in the scenario steps, there are used the highlighted tokens:

GET (corresponds to RequestMethod.GET)
INTERNAL_SERVER_ERROR (corresponds to HttpStatus.INTERNAL_SERVER_ERROR)

In the scenario steps they are referenced in the following manner:

WireMockApiSteps.java

  @Then("I have made {int} {requestMethod} calls made towards Github API {string} resource")
  public void checkNumberOfApiCalls(int count, RequestMethod requestMethod, String uri) {
    // ...

UserSteps.java

  @Then("I will receive an {httpStatus} response status instead of the user details")
  public void checkErroneousCall(HttpStatus httpStatus) {
    // ...

Gherkin allows the registration of custom parameter types. See below the corresponding code for registering the custom parameter types:

ParameterTypes.java

    typeRegistry.defineParameterType(new ParameterType<>(
        "requestMethod", // name
        "GET|POST|PUT|DELETE", // regexp
        RequestMethod.class, // type
        (io.cucumber.cucumberexpressions.Transformer<RequestMethod>) s -> {
          RequestMethod requestMethod;
          if ("GET".equals(s)) {
            requestMethod = RequestMethod.GET;
          } else if ("POST".equals(s)) {
            requestMethod = RequestMethod.POST;
          } else if ("PUT".equals(s)) {
            requestMethod = RequestMethod.PUT;
          } else if ("DELETE".equals(s)) {
            requestMethod = RequestMethod.DELETE;
          } else {
            throw new IllegalArgumentException("Unknown value " + s + " for RequestMethod");
          }
          return requestMethod;
        }
    ));

    var regexpHttpStatus = Arrays.stream(HttpStatus.values()).map(Enum::name)
        .collect(Collectors.joining("|"));
    typeRegistry.defineParameterType(new ParameterType<>(
        "httpStatus", // name
        regexpHttpStatus, // regex
        HttpStatus.class, // type
        (io.cucumber.cucumberexpressions.Transformer<HttpStatus>) HttpStatus::valueOf
    ));

Data tables

Data tables is a feature of Gherkin through which can be elegantly wrapped related values as objects in order to be passed the step definitions:

AccuracyCases.feature

    Given I have configured the responses for the Github API
      | uri               | httpStatus | payloadFile                      |
      | /users/findinpath | 200        | api/github/users/findinpath.json |

WireMockApiSteps.java

  @Given("I have configured the responses for the Github API")
  public void configureApiResponses(List<GithubApiResponse> responseList) {
    // ...
  }

  // ...

  public static class GithubApiResponse {

    private String uri;
    private int httpStatus;
    private String payloadFile;

    // ...

Cucumber Spring Integration

Cucumber integration with spring framework has a few tweaks are worth mentioning in the lines below.

The CommonSteps.java class is the only one annotated with @SpringBootTest annotation. This is one of the limitations imposed by the Cucumber framework in the ingration with the spring framework.

@SpringBootTest(classes = SpringDemoTestApplication.class)
@ActiveProfiles("test")
public class CommonSteps {
  //...
}

The step definition classes, the so-called “glue” to the test code get their dependendant spring beans injected by using the @Autowired annotation on the constructor of the step class. Below is presented the code of UserSteps class constructor for showcasing this particularity:

  @Autowired
  public UserSteps(UsersEndpoint usersEndpoint,
      UserSharedScenarioData userSharedScenarioData) {
    this.usersEndpoint = usersEndpoint;
    this.userSharedScenarioData = userSharedScenarioData;
  }

WireMock

The library WireMock is being used in the tests in order to be able to mock the Github API.

Particularly useful, in case of failure tests, was the Stateful Behavior for being able to simulate failures and recoveries when calling the mock API for a specific endpoint. The relevant code from WireMockApiSteps.java for configuring the mocked Github API responses is presented below:

    Given I have configured the responses for the Github API
      | uri               | httpStatus | payloadFile                      |
      | /users/findinpath | 500        |                                  |
      | /users/findinpath | 200        | api/github/users/findinpath.json |

  @Given("I have configured the responses for the Github API")
  public void configureApiResponses(List<GithubApiResponse> responseList) {
    var server = wireMockGithubApi.getWireMockServer();

    var responsesByUriMap = responseList
        .stream()
        .collect(Collectors.groupingBy(githubApiResponse -> githubApiResponse.getUri().trim()));

    responsesByUriMap.forEach((uri, uriResponseList) -> {
      for (int index = 0; index < uriResponseList.size(); index++) {
        var scenarioState =
            (index == 0) ? Scenario.STARTED : "Attempt " + (index + 1) + " for " + uri;

        var scenarioName = uri;
        var githubApiResponse = uriResponseList.get(index);
        var scenarioMappingBuilder = WireMock
            .get(WireMock.urlEqualTo(uri))
            .inScenario(scenarioName)
            .whenScenarioStateIs(scenarioState);
        if (index != uriResponseList.size() - 1) {
          scenarioMappingBuilder = scenarioMappingBuilder
              .willSetStateTo("Attempt " + (index + 2) + " for " + uri);
        }
        if (githubApiResponse.getHttpStatus() == HttpStatus.OK.value()) {
          var response = WireMock.aResponse()
              .withHeader("Content-Type", "application/json")
              .withStatus(githubApiResponse.getHttpStatus());
          if (githubApiResponse.getPayloadFile() != null) {
            response.withBodyFile(githubApiResponse.getPayloadFile());
          }
          server.stubFor(
              scenarioMappingBuilder
                  .willReturn(response)
          );
        } else {
          server.stubFor(
              scenarioMappingBuilder
                  .willReturn(
                      WireMock.aResponse()
                          .withHeader("Content-Type", "application/json")
                          .withStatus(githubApiResponse.getHttpStatus())
                  )
          );
        }
      }
    });
  }

Also very useful has proved to be the ability to browse through the requests made towards the WireMock server.

    And I have made 2 GET calls made towards Github API "/users/findinpath" resource
    But I have a backoff delay between GET requests 1 and 2 made towards Github API "/users/findinpath" resource

Check more details on how to browse through the requests reaching WireMock Server on the blog post

Browse the wiremock requests https://t.co/ze1neOa1CK
— findinpath (@findinpath) November 14, 2019

Spring-retry

The spring-retry topic has already been covered in a previous post. Check it out for seeing how to layer several AOP aspects on top of each other in order to get in-depth metrics (including retries) on how much time external API call takes.

Add metrics to spring-retry functionality https://t.co/LYCuunrztT
— findinpath (@findinpath) November 20, 2019

The spring-retry functionality is configured in the github-api-aop-config.xml file:

  <aop:config>
    <aop:pointcut id="github-api-calls"
      expression="execution(* com.findinpath.api.github.UsersEndpoint.*(..))  "/>

    <aop:advisor pointcut-ref="github-api-calls"
      advice-ref="githubApiRetryAdvice" order="1"/>

  </aop:config>

Conclusions

As already mentioned in the Cucumber documentation the main benefits of using this libray for testing a software project are:

Encouraging collaboration across roles to build shared understanding of the the problem to be solved

Working in rapid, small iterations to increase feedback and the flow of value

Producing system documentation that is automatically checked against the system’s behaviour

Nevertheless, there are some drawbacks that have to be taken into account before integrating Cucumber in your project:

Poorly written tests can easily increase test-maintenance cost
Cucumber is based (at the moment of this writing) on JUnit 4. Junit 5 will be supported with the upcoming release of Cucumber 5 (see on Github the Add JUnit 5 Support issue)
Cucumber tests are executed in a single-threaded fashion. This may be an incovenience for projects having a lot of tests.

With all these things said, Cucumber is a very viable alternative for end to end tests because it enables collaboration across all the roles in the project (software engineer, quality assurance, project management, requirements analyst) to build shared understanding of the the problem to be solve.

Spring retry metrics

Wed, 20 Nov 2019 00:00:00 GMT

This demo showcases how to add metrics on the spring-retry functionality with the micrometer library.

The project spring-retry offers both declarative and imperative retry support for spring applications.

Below is presented a simple code snippet to get an idea about how the imperative retry support can be configured for a retriable functionality:

RetryTemplate template = RetryTemplate.builder()
				.maxAttempts(3)
				.fixedBackoff(1000)
				.retryOn(RemoteAccessException.class)
				.build();

template.execute(ctx -> {
    // ... do something
});

Coupling together spring AOP and spring-retry gives the ability to perform retries on the public method calls that correspond to a pointcut. This could come pretty handy when dealing with a class or a set of classes which expose the API of an external service.

Due to various reasons, every now and then it happens that an API call doesn’t succeed, but when trying again everything works just fine.

This post showcases the ability to collect metrics via micrometer library on:

how many calls to the external API were successful
how many calls to the external API were retried once, twice, three times
how many calls to the external API have failed and with which exception type

The project spring-retry-metrics is referenced throughout this blog post in order to have a concrete implementation of the concepts presented.

For the sake of a concrete example of an external API, the project spring-retry-metrics makes use of a slimmed mocked version of the Github API. No actual calls towards Github API are made during the tests of this project.

Below is presented a test scenario from the project for showing at work both the spring-retry functionality (the Github repository details are retrieved successfully, even though the first call to the Github API fails) and the metrics of the spring-retry functionality on the methods advised via spring AOP for the class com.findinpath.github.api.GithubApi

GithubApiRetryTest.java

  @Test
  public void firstApiOperationCallFails() throws Exception {
    var blogRepository = new GithubRepository(BLOG_REPOSITORY_NAME,
        new URL("https://github.com/findinpath/blog"),
        false);

    when(restClient.getForEntity(
        eq(API_URL + "orgs/" + ORGANISATION_NAME + "/repos/" + BLOG_REPOSITORY_NAME),
        eq(GithubRepository.class))
    )
    .thenThrow(new IllegalStateException("Internal server error"))
    .thenReturn(blogRepository);

    var repository = githubApi.getOrganisationRepository(ORGANISATION_NAME, BLOG_REPOSITORY_NAME);
    assertThat(repository, equalTo(blogRepository));

    // check that the metrics are collected as expected
    var meters = meterRegistry.getMeters();
    var githubApiTimer = getExactlyOneMeter(meters, API_METRIC_NAME,
        Timer.class,
        Tag.of("exception", "none"),
        Tag.of("class", "GithubApi"),
        Tag.of("method", "getOrganisationRepository"));
    var githubApiExceptionTimer = getExactlyOneMeter(meters, API_METRIC_NAME,
        Timer.class,
        Tag.of("exception", "IllegalStateException"),
        Tag.of("class", "GithubApi"),
        Tag.of("method", "getOrganisationRepository"));
    var githubApiRetryTimer = getExactlyOneMeter(meters, API_RETRY_METRIC_NAME,
        Timer.class,
        Tag.of("class", "GithubApi"),
        Tag.of("method", "getOrganisationRepository"));

    assertThat(githubApiTimer.count(), equalTo(1L));
    assertThat(githubApiExceptionTimer.count(), equalTo(1L));
    assertThat(githubApiRetryTimer.count(), equalTo(1L));
    assertThat(githubApiRetryTimer.max(TimeUnit.MILLISECONDS),
        greaterThan(githubApiTimer.max(TimeUnit.MILLISECONDS)));
    assertThat(githubApiRetryTimer.max(TimeUnit.MILLISECONDS),
        greaterThan((double) GithubApiRetryTest.TestConfiguration.INITIAL_BACKOFF_TIME));

    var githubApiRetriesCounter = getExactlyOneMeter(meters, API_METRIC_NAME + "_retries",
        Counter.class,
        Tag.of(MicrometerRetryListenerSupport.CLASS_TAG_NAME, "GithubApi"),
        Tag.of(MicrometerRetryListenerSupport.METHOD_TAG_NAME, "getOrganisationRepository"),
        Tag.of(MicrometerRetryListenerSupport.RETRY_TAG_NAME, "1"),
        Tag.of(MicrometerRetryListenerSupport.EXCEPTION_TAG_NAME, "IllegalStateException"));
    assertThat(githubApiRetriesCounter.count(), equalTo((double) 1));

  }

Spring AOP configuration

The following Spring AOP configuration serves for the following purposes:

times how long each Github API call takes to complete including internal retries (in case that the first,second, … call to the API doesn’t succeed)
adds spring-retry functionality on all exposed Github API calls
times how long each Github API call takes to complete

github-api-aop-config.xml

  <aop:config>
    <aop:pointcut id="github-api-calls"
      expression="execution(* com.findinpath.github.api.GithubApi.*(..))  "/>

    <!--
    the githubApiRetriesIncludedTimedAdvice advice wraps
    the githubApiRetryAdvice advice and this it can provide
    timing information for the duration of the API call
    including retries
     -->
    <aop:advisor pointcut-ref="github-api-calls"
      advice-ref="githubApiRetriesIncludedTimedAdvice" order="1"/>
    <aop:advisor pointcut-ref="github-api-calls"
      advice-ref="githubApiRetryAdvice" order="2"/>
    <!--
    timing advice for each (retries are not taken into account)
    of the Github API calls.
    -->
    <aop:advisor pointcut-ref="github-api-calls"
      advice-ref="githubApiTimedAdvice" order="3"/>

  </aop:config>

Notice on the configuration above that several Spring AOP advisors are stacked like ognion layers on top of each other. The precedence of the advisor is determined by the order parameter of the advisor.

Spring-retry enhancements

At the time of this writing, spring-retry library in the version 1.2.4.RELEASE doesn’t provide the ability to retrieve information about the invoked method (see org.aopalliance.intercept.MethodInvocation) in the spring AOP context. See the corresponding Github issue

This limitation doesn’t allow to access the invoked class and method name in the concrete implementations of the org.springframework.retry.listener.RetryListenerSupport defined for the spring-retry’s RetryTemplate. What this means is that when a spring-retry failure occurs on a specific API call, we only would know in the monitoring that one of the API calls failed, but not exactly which. For our concrete example, if Github API has several exposed endpoints, it would be surely important to know via monitoring dashboards which one of these endpoints is causing failures in our application.

In order to enhance the metrics collected for the methods advised with spring-retry functionality, the project spring-retry-metrics has performed a little enhancement on the default spring-retry functionality by adding the MethodInvocation of the advised method to the retry context.

See the code source for the class org.springframework.retry.interceptor.MethodInvocationRetryOperationsInterceptor for more details.

Run the project

Run the command

mvn clean install

for executing the tests from this project.

End to end tests for KafkaListener

Mon, 18 Nov 2019 00:00:00 GMT

This post shows how the KafkaListener belonging to the spring-kafka library can be tested in an end-to-end fashion for both json and avro mesages.

The spring-kafka comes with a few testing utilities, but it doesn’t provide any utilities for testing the methods annotated with the KafkaListener annotation. Moreover, it makes use of an embedded Apache Kafka broker, instead of dockerized Apache Kafka container image artifacts.

This post concentrates on the concepts implemented in the project kafkalistener-e2e-test for dealing with end-to-end-testing for the methods annotated with the KafkaListener annotation.

The testcontainers library is employed for spawning before the tests a complete Confluent ecosystem of docker container images for artifacts related to Apache Kafka:

Apache Kafka
Apache Zookeeper
Confluent Schema Registry

By using versions for the container images that correspond to the Apache Kafka ecosystem from the production environment, there is simulated an environment which is very close to the one running in the production. This particularity gives a high relevance to the integration/ end-to-end tests for the kafka listener functionality.

It is very important to have the ability to perform end-to-end tests in a throwaway dockerized environment because there can be executed with a high certainty common scenarios that the kafka listener service is supposed to handle as part of its contract.

End to End Test setup

As mentioned previously, by employing the testcontainers library an entire Apache Kafka ecosystem will be spawned at the beginning of the tests. Check out the implementation related to testcontainers in the project kafkalistener-e2e-test for seeing how the Apache Kafka ecosystem artifacts are configured to work together for setting up the testing enviroment for the end to end tests.

//KafkaTestContainers.java

public KafkaTestContainers() throws IOException {
  this.network = Network.newNetwork();
  this.zookeeperContainer = new ZookeeperContainer()
      .withNetwork(network);
  this.kafkaContainer = new KafkaContainer(zookeeperContainer.getZookeeperConnect())
      .withNetwork(network);
  this.schemaRegistryContainer = new SchemaRegistryContainer(
      zookeeperContainer.getZookeeperConnect())
      .withNetwork(network);

  Startables
      .deepStart(Stream.of(zookeeperContainer, kafkaContainer, schemaRegistryContainer))
      .join();

}

Once the Apache Kafka ecosystem is up and running, the topics necessary for the end-to-end tests are created and the AVRO types are registered to Confluent Schema Registry docker container.

// KafkaDockerConfiguration.java

@Bean
public KafkaTestContainers kafkaTestContainers(
    @Value("${kafka.userBookmarkEventsJson.topic}") String userBookmarkEventJsonTopic,
    @Value("${kafka.userBookmarkEventsAvro.topic}") String userBookmarkEventAvroTopic
) throws Exception {
  var kafkaTestContainers = new KafkaTestContainers();

  createTopics(kafkaTestContainers, userBookmarkEventJsonTopic, userBookmarkEventAvroTopic);
  registerSchemaRegistryTypes(kafkaTestContainers.getSchemaRegistryContainer());
  return kafkaTestContainers;
}

After this setup, the rest of the spring beans from Spring’s dependency injection container (including the kafka listeners) are initialized and at this time there can be executed end-to-end tests.

// UserBookmarkEventJsonListenerTest.java

@MockBean
private UserBookmarkEventService userBookmarkEventService;

@Test
public void demo() {
  // GIVEN
  var userId = UUID.randomUUID().toString();
  var url = "https://findinpath.com";
  UserBookmarkEvent userBookmarkEvent = new UserBookmarkEvent(userId, url,
      Instant.now().toEpochMilli());

  // WHEN
  writeToTopic(userBookmarkEventJsonTopic, userBookmarkEvent);

  // THEN
  var argumentCaptor = ArgumentCaptor.forClass(UserBookmarkEvent.class);
  verify(userBookmarkEventService, timeout(10_000)).ingest(argumentCaptor.capture());
  UserBookmarkEvent capturedUserBookmarkEvent = argumentCaptor.getValue();
  assertThat(userBookmarkEvent, equalTo(capturedUserBookmarkEvent));
}

The demo test is quite straightforward, because it only concentrates to make sure that the service responsible of the business logic of handling the message is being called. Nevertheless, such a test ensures that the correct service is being called in the kafka listener and also that the message sent to the kafka topic is correctly deserialized.

Limitations

Compared to the tests in which the tests in which the KafkaConsumer can be manipulated directly in order to be able to reset the consumer offset after each test, spring-kafka hides the consumer instance inside the class org.springframework.kafka.listener.KafkaMessageListenerContainer.listenerConsumer with a private access. Even with extra motivation, when accessing the private field via Java Reflection, for resetting its offset, the operations on it will fail because multi-threaded access on the consumer is not supported (see org.apache.kafka.clients.consumer.KafkaConsumer.acquire method).

But even with the limitation of not being able to reset the consumer offset, it is still quite useful to ensure the fact that the right service is being called to handle Kafka message sent over the topic (otherwise said, regression test).

Source code

The proof of concept project kafkalistener-e2e-test offers two end-to-end sample tests:

com.findinpath.kafka.listener.UserBookmarkEventAvroListenerTest : for testing the consumption of messages serialized in AVRO format
com.findinpath.kafka.listener.UserBookmarkEventJsonListenerTest : for testing the consumption of messages serialized in JSON format

Run the command

mvn clean install

for executing the tests from this project.

Timing spring-data-cassandra repository methods

Fri, 15 Nov 2019 00:00:00 GMT

This post showcases how to make use of Spring AOP / AspectJ for timing the duration of both synchronous and asynchrnous methods belonging to spring data cassandra repositories.

Metrics in the data access layer of the application

Having an overview on the Prometheus for how each of the operations exposed by the Data Access Layer performs is a tremendous improvement for monitoring and tackling production issues on a live application.

To give a pretty common example, say that with a new release of the application one SELECT statement from a repository class gets changed that causes the application to run normal on the testing environment, but on the productive environment where there’s much more data to perform much worse than it did before. In that case after the release, the Quality of Service will drop to a significant amount, and the engineer responsible for the release will need to investigate what has changed for the worse since the last release.

This is where nowadays metric collector services (like Prometheus) shine and provide a lot of meaningful answers in such situations. It would obviously be very helpful to see in Grafana or even receive a Prometheus alert whether an operation or a set of operations performed by the application performs much more slowly than before.

Seeing a continuous increase of average duration or repeated failures on the data access layer for a certain operation from a repository in the data access layer of the application since the last release would help to identify very fast the problem and subsequently revert the change or fix the problem.

Timing the methods of spring data repositories

When using the io.micrometer.core.annotation.Timed annotation that comes with micrometer library in a spring-boot application there is obtained the emission of timing metrics for the methods annotated with this annotation.

When annotating the class with the @Timed annotation, all the public methods of the spring bean class will be timed.

This seems like a good solution, but very often it happens that through omission/refactoring in the code, the annotation is being removed by mistake from the class and therefore the valuable metrics and not emitted anymore to the micrometer’s MeterRegistry.

Another drawback of the io.micrometer.core.annotation.Timed annotation (and the io.micrometer.core.aop.TimedAspect which takes care of intercepting the methods annotated with Timed annotation) is that it doesn’t have support for dealing with asynchronous methods.

Through spring data cassandra utility classes:

org.springframework.data.cassandra.core.CassandraTemplate
org.springframework.data.cassandra.core.AsyncCassandraTemplate

there can be executed statements on the Cassandra database in both fashions, synchronous and asynchronous.

The project spring-data-cassandra-repository-methods-timing developed a custom AspectJ class for taking care of timing all the public methods exposed by the spring data repository bean classes from the project.

Spring data repository classes are considered any of the following:

the class is annotated with the @org.springframework.stereotype.Repository annotation
the class implements org.springframework.data.repository.Repository class

In a similar fashion to the micrometer’s io.micrometer.core.aop.TimedAspect, the AspectJ com.findinpath.aop.RepositoryTimerAspect class from the proof of concept project intercepts the methods of spring data repository beans from the project and emits timer information to micrometer’s io.micrometer.core.instrument.MeterRegistry.

The information get emitted to the micrometer metric named repository with the following tags:

.tags("class", className) 
.tags("method", methodName)
.tags("successful", successful)

The className can be on of the following:

the interface corresponding to the repository - see com.findinpath.repository.ConfigRepository
the concrete class annotated with @org.springframework.stereotype.Repository annotation - see com.findinpath.repository.UserBookmarkRepository

The successful flag states whether the call was or not successfully executed.

Dealing with asynchronous methods

As mentioned earlier, there can be executed asynchronous methods on Cassandra. The implementation of the asynchronous methods with spring data cassandra is straightforward (see com.findinpath.repository.UserBookmarkRepository for more details):

  public ListenableFuture<UserBookmark> saveAsync(UserBookmark userBookmark) {
    return asyncCassandraOperations.insert(userBookmark);
  }

A little trick needs therefor to be employed in the AspectJ RepositoryTimerAspectclass for timing such non-blocking methods. After retrieving the result from the target method invocation, a callback is being registered for emiting the metrics in both success and failure cases after the completion of the method:

    var asyncResult = (ListenableFuture) proceedingJoinPoint.proceed();
    asyncResult.addCallback(
        result -> emitMetrics(meterRegistry, sample, className, methodName, Optional.empty()),
        ex -> emitMetrics(meterRegistry, sample, className, methodName, Optional.of(ex)));
    return asyncResult;

Spring AOP

In case that it is needed for reference a Spring AOP implementation for timing the spring data repositories, the project source code provides also such an implementation, although commented, in order to avoid causing issues with the AspectJ `RepositoryTimerAspect.

See com.findinpath.config.RepositoryTimerConfiguration for details.

Demo

The project spring-data-cassandra-repository-methods-timing comes with a com.findinpath.DemoTest class which contains test scenarios for timing both kinds of spring data repositories:

the class is annotated with the @org.springframework.stereotype.Repository annotation
the class implements org.springframework.data.repository.Repository class

and also both synchronous and asynchronous methods.

Below are the measurements logged from one of the runs of the DemoTest:

The timer MeterId{name='repository', tags=[tag(class=ConfigRepository),tag(method=save),tag(successful=true)]} has max value: 176 ms and mean value: 25 ms
The timer MeterId{name='repository', tags=[tag(class=ConfigRepository),tag(method=findById),tag(successful=true)]} has max value: 22 ms and mean value: 8 ms

The timer MeterId{name='repository', tags=[tag(class=UserBookmarkRepository),tag(method=save),tag(successful=true)]} has max value: 32 ms and mean value: 9 ms
The timer MeterId{name='repository', tags=[tag(class=UserBookmarkRepository),tag(method=saveAsync),tag(successful=true)]} has max value: 33 ms and mean value: 8 ms
The timer MeterId{name='repository', tags=[tag(class=UserBookmarkRepository),tag(method=findLatestBookmarks),tag(successful=true)]} has max value: 12 ms and mean value: 7 ms
The timer MeterId{name='repository', tags=[tag(class=UserBookmarkRepository),tag(method=findLatestBookmarksAsync),tag(successful=true)]} has max value: 25 ms and mean value: 12 ms

The test is being executed against a throwaway Cassandra database container (through the help of the genius team formed from docker and the testcontainers library).

In order to keep the demo project simple to go through, no setup has been made for a web application that would expose operations to interact with Cassandra database. This is why in order to test the concepts showcased by the project, only the com.findinpath.DemoTest class is provided.

Simply use mvn clean install for trying out the project.

Cassandra schema migrations on application startup

Thu, 14 Nov 2019 00:00:00 GMT

This post shows how to setup a spring boot project in order to have the Cassandra CQL schema migrations securely performed on application startup through the help of the cassandra-migration library.

Database migrations are a topic that the software engineers have long struggled with. After doing a git pull for the latest changes on the application code, it happened quite often that the application didn’t work anymore on the local machine simply because a database table had to be created or altered manually on the local database.

Not that seldom production application crashes occurred during deployments due to the fact that the database engineer didn’t run all the database scripts corresponding to the new application release.

In order to address the before mentioned problems, in the relational database world, the SQL schema migrations have been integrated in the application code and the new schema migrations are executed on the database while the application is being started.

The libraries:

are used extensively in the Java world for performing database schema migrations on the application startup. As a side note, these libraries offer the possibility to dryrun and generate all the SQL scripts relevant for the new productive application release into a .sql file which can be then reviewed and executed manually by a database system administrator.

Due to the fact that Cassandra database is not a relational database, flyway doesn’t offer support for performing schema migrations on top of Cassandra.can’t be There is actually an Github issue that dates since 2014 on flyway, which eventually sparked the creation of the cassandra-migration library.

This post shows how to integrate the cassandra-migration library in a spring boot application in order not only to perform the schema migrations on the application startup, but also when the test environment is being setup.

In this way, both the productive and the test code use the same database schema.

Separation of concerns

As already mentioned, the schema migrations are performed only during application startup phase. This kind of database statements (CREATE, ALTER, DROP) are to be performed by a database user which obviously is thought only for schema migrations purposes.

On the other hand, when thinking of security concerns for the application, the database user required for executing database statements required for covering the application functionality (SELECT, INSERT, DELETE) should have more restrictive permissions. In this manner, there can be avoided catastrophic changes on the database (DROP , TRUNCATE) that can occur through software bugs or application attacks.

Therefor it makes very much sense to have two different Cassandra database users in the application:

a migration user dealing solely with CQL migrations on the application startup
an application user dealing with the CQL queries required for offering the functionality exposed by the application

Spring-boot integration

The org.springframework.boot.autoconfigure.data.cassandra.CassandraDataAutoConfiguration and org.cognitor.cassandra.migration.spring.CassandraMigrationAutoConfiguration spring boot auto-configuration classes are based on the presumption that only one database user is used for both application and migration database queries. For this reason, they need to be excluded from being taken into consideration when initializing the spring dependency injection bean container.

@SpringBootApplication(exclude = {CassandraDataAutoConfiguration.class,
    CassandraMigrationAutoConfiguration.class})
public class DemoApplication {
  //...
}

The project sample code provided with this post creates explicitly two different clusters:

applicationCluster
migrationCluster

@Configuration
@Profile("!test")
public class CassandraConfiguration {

  @Bean
  @CassandraMigrationCluster
  public CassandraClusterFactoryBean migrationCluster(
      @Value("${cassandra.contact.points}") String contactPoints,
      @Value("${cassandra.migration.username}") String username,
      @Value("${cassandra.migration.password}") String password) {
    PoolingOptions poolingOptions = new PoolingOptions()
        .setConnectionsPerHost(HostDistance.LOCAL, 1, 1)
        .setConnectionsPerHost(HostDistance.REMOTE, 1, 1)
        .setMaxRequestsPerConnection(HostDistance.LOCAL, 1)
        .setMaxRequestsPerConnection(HostDistance.REMOTE, 1);

    return createCassandraClusterFactoryBean(poolingOptions, contactPoints, username, password);
  }

  @Bean(name = "applicationCluster")
  public CassandraClusterFactoryBean applicationCluster(
      @Value("${cassandra.contact.points}") String contactPoints,
      @Value("${cassandra.application.username}") String username,
      @Value("${cassandra.application.password}") String password) {
    PoolingOptions poolingOptions = new PoolingOptions();
    return createCassandraClusterFactoryBean(poolingOptions, contactPoints, username, password);
  }

The applicationCluster is being used for building the CassandraTemplate bean which will subsequently power all the spring data cassandra repository classes.

public class CassandraConfiguration {
  ....

  @Bean
  public CassandraSessionFactoryBean cassandraSessionFactoryBean(
      @Qualifier("applicationCluster") CassandraClusterFactoryBean applicationCluster,
      @Value("${cassandra.application.keyspaceName}") String keyspaceName) {

    Cluster cluster = applicationCluster.getObject();
    CassandraSessionFactoryBean session = new CassandraSessionFactoryBean();
    session.setCluster(cluster);
    session.setKeyspaceName(keyspaceName);
    session.setConverter(cassandraConverter());
    session.setSchemaAction(SchemaAction.NONE);

    return session;
  }

  @Bean
  public CassandraOperations cassandraTemplate(
      CassandraSessionFactoryBean cassandraSessionFactoryBean) {
    return new CassandraTemplate(cassandraSessionFactoryBean.getObject());
  }

The migrationCluster cluster on the other hand is being used only during the application startup for performing the Cassandra schema migrations with the help of cassandra-migration library.

@Configuration
@EnableConfigurationProperties({CassandraMigrationConfigurationProperties.class})
@ConditionalOnClass({Cluster.class})
public class CassandraMigrationConfiguration extends CassandraMigrationAutoConfiguration {
  
  // ...

  @Bean(initMethod = "migrate")
  @ConditionalOnMissingBean(MigrationTask.class)
  public MigrationTask migrationTask(
      @CassandraMigrationCluster ObjectProvider<Cluster> migrationClusterProvider) {
    Cluster migrationCluster = migrationClusterProvider.getIfAvailable();
    return super.migrationTask(migrationCluster);
  }

  // ...

With the configurations listed above the secure integration of the cassandra-migration library in the spring boot application should be almost complete.

All what is left, are the actual CQL migration files that come into the directory denoted by ${cassandra.migration.scriptLocation} environment property.

NOTE that each migration file should contain exactly one CQL statement. Therefor, in case that an application feature needs more database changes, they must be split to different files. Nevertheless, the naming of the migration files is pretty liberal, so the files can look like this:

0080_awesome_feature_create_bookmarks_table.cql
0081_awesome_feature_alter_users_table.cql

A word about testing

It would be ideal to have a throw-away database instance which has the same schema as the productive database that is being made available at the beginning of the tests . When using testcontainers library together with cassandra-migration this feat of quality assurance engineering is easily achieved.

Below is presented a relevant snippet from the Cassandra configuration class used for test purposes:

@Configuration
public class CassandraDockerConfiguration {
  // ...

  @Bean
  public CassandraContainer cassandraContainer() {
    var cassandraContainer = new CassandraContainer(CassandraContainer.IMAGE
        + CASSANDRA_DOCKER_IMAGE_VERSION);
    cassandraContainer.start();

    loadTestData(cassandraContainer);
    return cassandraContainer;
  }

  @Bean("applicationCluster")
  public CassandraClusterFactoryBean applicationCluster(CassandraContainer cassandraContainer) {
    final CassandraClusterFactoryBean cluster = new CassandraClusterFactoryBean();

    cluster.setContactPoints(cassandraContainer.getContainerIpAddress());
    cluster.setPort(cassandraContainer.getFirstMappedPort());
    return cluster;
  }

  @Bean("migrationCluster")
  @CassandraMigrationCluster
  public CassandraClusterFactoryBean migrationCluster(CassandraContainer cassandraContainer) {
    final CassandraClusterFactoryBean cluster = new CassandraClusterFactoryBean();

    cluster.setContactPoints(cassandraContainer.getContainerIpAddress());
    cluster.setPort(cassandraContainer.getFirstMappedPort());
    return cluster;
  }
  
  // ...

The setup of the tests is done afterwards pretty straightforward:

@SpringBootTest
@ActiveProfiles("test")
@TestPropertySource(properties = {"cassandra.migration.keyspaceName = demo",
    "cassandra.migration.scriptLocation: cassandra/migration"})
public class DemoTest {
  
  // ...
}

Runnable sample code

The java project cassandra-migration-spring-boot-demo provides a runnable example on how to perform the separation of concerns between the Cassandra application and migration users. The project also makes use of testcontainers library allowing therefor the repository tests to be executed against a real (and not embedded) Cassandra database.

Simply use mvn clean install for trying out the project.

Browse the requests made towards WireMock

Wed, 13 Nov 2019 00:00:00 GMT

This post showcases how to browse/filter the requests made towards Wiremock.

When doing integration tests that involve one or more external APIs, it is worth considering a mock server for the job.

Some of the advantages of using such a server is that it:

always performs as configured (even the delay for a API response can be in detail configured)
doesn’t have downtimes while the tests are being executed

Obviously, there are downsides when using a mock server for any testing scenario. It is advisable that for end to end tests to use the actual external APIs instead of the mocked versions of them in order to avoid discovering regression issues only in production.

WireMock is one of the best libraries that act as a simulator for HTTP-based APIs. It can be executed from a Junit test, as well as a standalone application.

WireMock offers the ability to browse through the requests made towards it and on top of this functionality there can be made assertions. This functionality can come in handy in several test scenarios.

Verify that a specific API endpoint has been called only once

wiremockServer.verify(1, postRequestedFor(urlEqualTo("api/order")));

Verify the order in which the API calls are made

var paymentRequests = wiremockServer.findAll(postRequestedFor(urlMatching("/api/payment")));
var shipmentRequests = wiremockServer.findAll(postRequestedFor(urlMatching("/api/shipment")));
// shipment shouldn't happen before payment
assertThat(shipmentRequests.get(0).getLoggedDate(), greaterThan(paymentRequests.get(0).getLoggedDate());

In case that a certain test needs a more complex handling (e.g. : callbacks in the test code) for specific requests, WireMock also provides a series of instruments that can be used to extend its basic functionality:

com.github.tomakehurst.wiremock.extension.requestfilter.RequestFilter
com.github.tomakehurst.wiremock.extension.PostServeAction
com.github.tomakehurst.wiremock.extension.ResponseDefinitionTransformer

Complex needs in the tests can be covered when registering a concrete implementation of the classes above as an extension for the WireMock server.

The java project wiremock-recorded-request-timestamp provides a runnable example on how to retrieve the logged date of the requests made towards the WireMock server.

Simply use mvn clean install for trying out the project.

Exchange Kafka avro messages in testcontainers ecosystem

Thu, 07 Nov 2019 00:00:00 GMT

This post showcases a manner on which to ensure the quality of Apache Kafka producers & consumers that exchange avro messages in a dockerized environment.

The library testcontainers can be used for providing lightweight, throw-away instances of docker containers corresponding to the artifacts of the Apache Kafka ecosystem:

Apache Zookeeper
Apache Kafka
Confluent Schema Registry

In this fashion, there can be tested complex integration scenarios which would otherwise involve the presence of an actual Apache Kafka testing environment. Simulating, for example, a schema version update for an avro event transported over Apache Kafka, can be tested therefor safely in an integration test without needing to occasionally manually test this operation.

The testcontainers library already offers a Kafka module for interacting with Apache Kafka, but there is not, at the moment, support on it for a Confluent Schema Registry container.

The project testcontainers-kafka-avro is a working prototype on how to setup and use in unit tests all the containers corresponding to the artifacts of the Apache Kafka ecosystem.

Setting up of the containers is done in the following manner:

  @BeforeAll
  public static void confluentSetup() throws Exception {
    network = Network.newNetwork();
    zookeeperContainer = new ZookeeperContainer()
        .withNetwork(network);
    kafkaContainer = new KafkaContainer(zookeeperContainer.getZookeeperConnect())
        .withNetwork(network);
    schemaRegistryContainer = new SchemaRegistryContainer(zookeeperContainer.getZookeeperConnect())
        .withNetwork(network);

    Startables
        .deepStart(Stream.of(zookeeperContainer, kafkaContainer, schemaRegistryContainer))
        .join();

    // ...
  }

The beauty of working with the testcontainers library is that there (considering that docker installed on the test machine) is nothing related to Apache Kafka to be installed on the test machine. Simply by running mvn clean install everything is being automatically setup on docker and at the end of the test, the containers needed by it will be disposed.

As a side note, the containers used do not use the default ports exposed by default in their corresponding artifacts (e.g. : Apache Zookeeper 2181, Apache Kafka 9092, Confluent Schema Registry 8081), but rather free ports available on the test/build machine. In this fashion there are avoided possible conflicts with already running services on the test/build machine.

The creation of the Apache Kafka topic needed in the test, as well as the registration of the avro types in the Confluent Schema Registry is done once while setting up the Confluent environment.

The test showcased in the project shows how a Kafka Producer writes a BookmarkEvent avro event to the BookmarkEvents topic from which a Kafka Consumer reads it and verifies that the message received is the same as the message sent.

  @Test
  public void demo() {
    final UUID userUuid = UUID.randomUUID();
    final BookmarkEvent bookmarkEvent = new BookmarkEvent(userUuid.toString(), URL,
        Instant.now().toEpochMilli());

    produce(TOPIC, bookmarkEvent);
    LOGGER.info(
        String.format("Successfully sent 1 BookmarkEvent message to the topic called %s", TOPIC));

    var consumerRecords = dumpTopic(TOPIC, 1, POLL_TIMEOUT_MS);
    LOGGER.info(String.format("Retrieved %d consumer records from the topic %s",
        consumerRecords.size(), TOPIC));

    assertThat(consumerRecords.size(), equalTo(1));
    assertThat(consumerRecords.get(0).key(), equalTo(bookmarkEvent.getUserUuid()));
    assertThat(consumerRecords.get(0).value(), equalTo(bookmarkEvent));
  }

Even though this demo showcases a rather naive usecase, the setup shown above for Apache Kafka ecosystem can be used for complex quality assurance scenarios like:

simulating how are handled by the producer/consumer the schema version updates of the avro messages
investigate how the consumer/producer functions when the schema registry is temporarily not available (when pausing the schema registry docker container)

Feel free to integrate the Apache Kafka ecosystem testcontainers classes in your project to ensure the accuracy of your Apache Kafka avro producers & consumers.

Retrieve the distinct partition keys of a Cassandra table

Tue, 05 Nov 2019 00:00:00 GMT

Selecting the distinct partition keys of a Cassandra table is very straightforward when performing a CQL query to retrieve the first page of DISTINCT partition keys of the Cassandra table, but in order to select all the distinct partition keys of the table without needing to retrieve all the rows of the table the TOKEN Cassandra function needs to be employed.

Let’s consider for the sake of having a concrete example to work on that we store the bookmarks made by the users in a Cassandra table.

CREATE TABLE DEMO.USER_BOOKMARKS (
	user_id UUID,
	timestamp TIMEUUID,
	url VARCHAR,
	PRIMARY KEY (user_id, timestamp)
) WITH CLUSTERING ORDER BY (timestamp DESC)

When the following query:

SELECT DISTINCT user_id
FROM demo.user_bookmarks
LIMIT 1000

is executed, it will provide the first page containing 1000 distinct partition keys of the table demo.user_bookmarks.

For selecting the next page, an extra filter based the TOKEN function has to be employed in order to retrieve the next page of distinct partition keys which all have the token greater as the token of the last user_id selected on the previous page.

SELECT DISTINCT user_id
FROM demo.user_bookmarks
WHERE TOKEN(user_id) > TOKEN(userIds[999]) 
LIMIT 1000

The corresponding java implementation looks like this:

private ResultSet getDistinctUserIdsBatch(Session session, Optional<UUID> lastUserIdProcessed,
		int batchSize) {
	var batchedDistinctUserSelect = QueryBuilder.select(USER_ID_COLUMN_NAME)
			.distinct()
			.from(DEMO_KEYSPACE_NAME, USER_BOOKMARKS_TABLE_NAME);

	lastUserIdProcessed.ifPresent(
			uuid -> batchedDistinctUserSelect.where(gt(token(USER_ID_COLUMN_NAME), token(uuid))));

	batchedDistinctUserSelect.limit(batchSize);

	return session.execute(batchedDistinctUserSelect);
}

The Cassandra TOKEN documentation is a good starter to get a feeling of what the TOKEN function does.

The java project cassandra-select-distinct-partition-keys provides a runnable test which starts a Cassandra docker container (via testcontainers library) fills it with data (the bookmarks made by a list of random users) and selects the distinct user identifiers.

Simply use mvn clean install for trying out the project.

find in path

Parse JSON in Trino

Data Mesh showcase with dbt-trino

Introduction

Data Mesh

jaffle_shop Data Mesh

Demo

dbt-trino

Conclusion

Inline Trino integer ranges

Use the memory connector and fill the table

Use an inline table

Unnest a generated sequence

Inline the range via UNION ALL

Inline the range via WITH RECURSIVE clause

Feedback

Infer deletions in parent-child ELT staging data

Historization of the user subscriptions

Infer deletions in the staging data

Demo

Feedback

Infer deletions in batched ELT staging data

Historization of the product availability

Infer deletions in the staging data

Simplified solution description

Demo

Feedback

dbt historized macro

Getting started with dbt

Demo

dbt historized macro

Data tranformation testing

Feedback

Feature toggles as a service

Unleash

Competition for Unleash

Demo

Articles related to feature toggles

Feedback

Price range aggregations

Static price ranges

Use two queries to get balanced price range aggregations

Collapse the buckets

Source Code

Feedback

Search Alerts

Technological stack

Architecture

Percolator

Immediate search alerts

Batched search alerts

Orchestrating batched messengers

Code

Open points

Elasticsearch percolator primer

How does the percolator actually work

Usage demo

Create news index

Populate news index with documents

Basic search operations on the news index

Create the news-notify index

Populate news-notify index

Percolate queries

Paging of the percolation results with search_after

Delete indexes

Resources

Code

JDBC Nested Set Sink Connector

Nested Set Model

Syncing nested set models over Apache Kafka

End to end synchronization over Apache Kafka Connect

Functionality overview

Testing

Source code

Installation notes

Sync tree database table over Apache Kafka

Nested Set Model

Syncing nested set models over Apache Kafka

Refresh the nested set model on the sink database

Caching

`jaffle_shop` Data Mesh

`dbt-trino`

Inline the range via `UNION ALL`

Inline the range via `WITH RECURSIVE` clause

Create `news` index

Populate `news` index with documents

Basic search operations on the `news` index

Create the `news-notify` index

Populate `news-notify` index

Paging of the percolation results with `search_after`