Apache Kafka and ESB in a Loosely Coupled Architecture: How to Reduce Costs, Guarantee Delivery, and Ensure Data Integrity in the IT Systems of Large Enterprises

Introducing a message broker partially reduces data loss, but does not eliminate it in some scenarios!

A broker is transport, not brains - all control and consumption logic lives inside the connected systems.

But integrations through a broker alone do not GUARANTEE delivery if the recipient is not ready, the format does not match, the logic has not been processed, the retention period has expired, or there is simply no receipt confirmation.

I should note separately that the broker has nothing to do with analyzing the causes of errors or overloads during peak days like Black Friday. Whether you have a broker or not, such errors or overloads often occur, and their cause remains unclear.

Kafka Peak load on systems is reduced Data loss is reduced

Asynchronous: systems do not need to stay online all the time to communicate; ESB with Kafka in the stack prevents data loss, significantly reduces system load, and makes business analytics (BI) easier.

With loose coupling through an ESB that includes Kafka, systems can be changed in a click.

IT costs are significantly reduced

Transferability - easy to hand over to another IT team

High data quality = simple business analytics

Data can be exchanged directly and transformed within systems.

It is important to account for transformation logic: if something changes in one system, another has to adapt. At first, this looks efficient: connect quickly via API, data starts flowing, great!

But without a unified integration layer and common standards, and without a loose-coupling approach, this starts to fall apart over time. The typical picture looks like this:

Many IT systems. Each has its own delivery and transformation logic.

Systems can be the source of dozens or even hundreds of entities with poor-quality data, which creates new errors and even more poor-quality entities.

Tools are introduced to clean up the data, such as MDM and PIM.

But they suppress the symptoms without removing the cause. As a result, one system must account for the transformation logic and connections of all the others.

That is strong coupling: a tangle of dependencies that is hard to untangle even within the team, let alone for external participants.

A closed-infrastructure effect emerges, where every change becomes a mini-project.

So simply implementing a message broker like Kafka, or even ESB, without a loose-coupling architecture is like taking painkillers for a brutal hangover.

It may get a little easier, but the root of the problem will still be untouched.

For two systems, an entity such as a product card can be a different set of data.

With ESB, it is easy to build data transformation tailored to each system.

This also reduces load and eliminates transfer errors caused by an incorrect data format.

Data is stored in processed form in the DWH and in its original form in the Data Lake.

Everything here is transparent and neatly organized.

The business intelligence (BI) system can easily generate any required report. And the BI system itself can be connected simply through a unified ESB integration layer.

Why will using ESB in general, or Kafka specifically, without loose coupling not produce the desired result?

All of the above is good, but to truly build an IT landscape where systems can be changed at the snap of a finger, IT costs can be significantly reduced, and support can be easily handed over to another team, loose coupling is needed.

Only this architecture delivers simple, accurate, and accessible business analytics.

This can be achieved by building loose coupling at both the architectural and organizational levels through a well-designed enterprise service bus (ESB), with Kafka used as the transport layer.

Here it is ineffective to simply introduce IT-layer solutions, because the digital copy of the business is drifting further and further from reality.

A gap emerges between how real processes actually work and what is reflected in IT.

You can endlessly change vendors, introduce new tools, and adopt trendy approaches.

But until there is identity between real and digital processes, all of this will keep producing errors and inconsistencies.

Solutions will be layered on top of these distortions, forming scar tissue with previous "solutions." And this digital erosion will keep compounding itself: the digital copy of the business will become less and less aligned with reality.

The conclusion is always the same: loose coupling is not an abstract idea, but a real factor that affects cost.

Below is a chart that clearly shows how the cost of maintaining an IT landscape grows without loose coupling. *

The IKEA Effect is a cognitive bias in which people overvalue things they made themselves, even if they are objectively worse.

We constantly see the same pattern: integrations are launched quickly at first through direct connections, which delivers fast results and a sense of efficiency.

But over time, the architecture starts to crumble: each new connection adds fragility, duplication, and dependence on specific systems and people.

Without a single owner who keeps the architecture in mind and manages the dependencies, the IT landscape turns into a tangled sweater: warm, but awkward, overheated, and full of holes.

It is like a good athlete training on their own, they will make progress.

But when an experienced coach appears, someone who can observe from the outside, correct technique, and manage the load, meaningful results and records follow.

The same is true in integrations: loose coupling does not happen on its own, it is achieved through systematic work and architectural support.