Kafka Integration into the Enterprise Infrastructure of a Project

Client

A major manufacturer and retailer of seating and cabinet furniture, mattresses, and sleep products.

Objective

For such a big company with multiple information systems, a point–to-point integration is not the best choice. The customer wanted a light data migration mechanism with a simple logic with a sole function of transmitting data from one system to another. At the same time, it was necessary to take into account the company's need for systems' relative independancy, so that it was possible to remove or replace one of the systems in such a way that other systems were not affected.

Solution

To cope with the customer's task, we've integrated Kafka into the enterprise infrastructure of the project. Kafka is a distributed system that allows users to exchange messages and provides permanent data storage capacities.

Kafka Integration

The system integration with the project infrastructure did not take much time: the system was deployed and the data producing (sending and receiving messages) and data consuming (message processing) were configured. To stream data between Kafka and other data systems it was necessary to add the code to these systems to interact with Kafka.

Kafka is connected to: Magento, Pimcore, Galactika ERP system, Creatio OMS (order processing system), ECM (content management system), a pricing service and payment services.

Web Interface Installation

To simplify Kafka management process, a standard UI interface has been installed. It allowed tracking the integration errors in real time: you can see which messages were not processed, add / remove topics (sequences), as well as test and monitor the sending of messages.

Now client doesn't need a developer for the troubleshooting — with the Kafka mechanism, even a manager will be able to analyse and solve problems!

Message Storage

Another advantage of Kafka is that the time of storing data in its system is unlimited. Even if the server crashes, you can restore the system from the backups, then open Kafka and go back to the message that was to be processed at the time of recovery from the copy, and continue the processing starting with this message. As a result, the application will process all messages suspended at the moment of server crash.