How to Choose the Right Type of Integration for Your IT Architecture

point-to-point, broker, ESB 24.4.2024

We examine the features, advantages, and disadvantages of three integration types: point-to-point, via a broker, and via ESB.

As a rule, they focus either on the technical details of building integrations or on the obvious pros and cons of each approach. In this article, KT.Team experts suggest looking at all three integration types from the perspective of their internal structure - this makes it possible to highlight not only the obvious but also the hidden advantages and disadvantages of each one.

Prefer to explore this material as a podcast?

The first and biggest drawback of this kind of integration is the strong mutual dependency between systems.

Let's continue our example with 1C and an online store.

For a user to place an order, the online store must receive the user's loyalty program status and address from 1C.

If 1C goes down or slows under heavy load, critical information will stop flowing to the online store.

Customers will not be able to place orders until 1C is back up.

To protect the business from such risks, the development team will keep inventing safeguards: setting flags, devising queues, reinventing asynchronous messaging between systems. In the end, the "inexpensive" direct integration will catch up with and overtake the cost of the "expensive" bus.

All of this has to be accounted for in the integration logic.

Third drawback: integrations overload the source system, even if it initially works with a small amount of data. For example, a CRM holds customer data that is constantly accessed by the online store, delivery, marketing applications, and so on.

The result is that load on the CRM grows many times over because of constant requests from other systems.

Fourth drawback: a change in any system triggers a cascade of rework across all related integrations and systems.

If you add a new field to a product card, you must update the logic in every system that consumes that card. And logic changes are not always painless: in especially critical cases, the only option is to roll back all changes.

To avoid overloading the warehouse system and sending the same data multiple times, developers may offer you an "elegant solution": let the WMS send everything only to 1C. Then 1C, in turn, sends to the online store not only its own data on items and orders, but also stock. What is wrong with this approach?

The accuracy of the stock data shown in the online store now depends on two systems working at once. 1C (or any intermediary system) ends up overloaded with data outside its core purpose.

In practice, we have seen even longer data enrichment chains, so this is not rare. And if something goes wrong in the destination system, an investigation is needed: where the data transfer failed, why it failed, and how to prevent repeats. The last, and in our view the most critical, drawback is that point-to-point integrations block or significantly slow down IT system updates.

If in three years you decide to replace that same CRM, you will have to redevelop and reproduce the logic of dozens of integrations — and not only the ones directly connected to the CRM.

Meanwhile, the company becomes hostage to the developers who know a lot about the existing logic.

Giving them up or switching the team to other tasks becomes almost impossible.

Developers themselves become hostages of their own knowledge: even on vacation, they have to stay reachable at all times, because who knows what could go wrong?

Data exchange becomes more controllable.

In the broker you can see which messages were sent out and which were delivered, check the logs, and review the event history. Second: source systems no longer need to find out whether the receiving system is available at the moment a message is sent. Their area of responsibility is limited to the logic of forming their own message queue.

Another advantage of brokers worth highlighting is contract support.

A contract is a check that a message follows basic rules, such as the presence of a specific field in the required format. Equally important is that this approach makes development more standardized.

It is also worth noting that data transfer through a broker is fast, and the broker is almost as fast as the quickest integration type, point-to-point.

The first important drawback is that using a broker adds load to the service that sends the message.

Imagine a situation: the broker is unavailable for some time, and no new messages are added to the queue during that period.

This logic puts extra load on the system, so there is a temptation to drop it.

But in that case, undelivered messages will sooner or later affect critical business processes.

The second drawback is that the target system becomes overloaded with unnecessary data.

The broker cannot choose which messages to pass on - it delivers the entire queue. For example, all assortment data or all stock data from the WMS.

But a system cannot "ask" the broker for filtered data.

It has to receive through integrations everything that the CRM and OMS (Order Management System) send, store it in its own databases, and use internal logic to separate the 5% of needed data from the 95% of excess data.

The third major drawback is contract hardening.

All participants in the process must agree on the format in which messages will be transmitted.

This removes flexibility and increases the labor costs of maintaining a single format.

A less obvious drawback is the lack of convenient and/or detailed history. For example, the Kafka broker can store historical data, but if something breaks, you will have to reload the entire message queue from the first historical record to the last.

It is impossible to run any queries on current statuses.

The problem is most acute when you need to exchange history with another team or perform a migration - extracting that information will be difficult.

This drawback brings another major one with it: the inability to perform proper reconciliation.

Systems load both old and new messages from the broker without distinguishing between them.

During reprocessing, a newer message may end up overwritten: for example, an order gets the status "created" instead of "delivered".

Such situations can be avoided, but that requires adding an intermediate reconciliation service or building reconciliation logic into the receiving system.

That is why reading data is much slower.

Using a broker can lead to proxying just as easily as direct integrations can.

To keep every system in the landscape from receiving redundant data, it is "easier" to route it through one system, normalize it there, and pass the normalized data on. We described above where this leads.

There is also a hidden drawback of a managerial nature. Every system has its own product owner.

But who owns the broker and the integrations tied to it?

Most often, it is the technical department that takes on this role.

For the other departments, this is an obvious plus (less responsibility), but in that case none of the stakeholders will feel they have control over the infrastructure.

The basic structure of ESB integrations helps explain where their advantages come from.

This is the only integration approach where all three components are separated and each performs strictly its own function: systems generate and consume data, the ETL layer delivers it to the right destination in the right form, and the storage (logically enough) stores the data.

One result of this is, for example, reduced load on services, since they contain no integration logic.

The service's job is to provide access to data via API if it is a modern service, FTP, or any other exchange method available in the service. The ETL layer will retrieve the data itself and pass it on to the warehouse.

The service "thinks" only about keeping all the information it is responsible for up to date and complete.

The second advantage follows from the first: the source system does not need to know the technical requirements of the data-consuming system.

Data format, transfer speed, update frequency, all of this logic is contained in the ETL layer.

Moreover, you can control the request rate: how many packets per unit of time are delivered to the service.

The bus sends the service only the data it actually needs - just build that logic into the connector.

A properly built ESB integration implies fault tolerance of the entire IT architecture.

Services do not exchange data directly: the ETL layer saves it to dedicated storage and retrieves the current version from there when needed. For example, if your bonus management system is temporarily unavailable, it will not affect order placement in the online store in any way.

The online store will get the bonus points and personal discount data from the data warehouse.

When the bonus management system comes back online, the connector will inform it about redeemed points and new orders so it can recalculate personal discounts for repeat customers. In a broader sense, using an ESB provides flexibility and resilience to cascading changes. Imagine you decided to replace a foreign CRM with a CIS one.

If you choose point-to-point integrations, first you will have to rebuild the point-to-point integrations between the CRM and other systems, and second, change the integrations of other systems that use CRM data. With ESB, you only need to build the flows to and from the CRM.

The flows of consumer systems remain unchanged.

It may seem that using an intermediate layer reduces speed.

In practice, however, the exchange speed is very high: tens, hundreds, thousands of messages are delivered within 400-500 milliseconds! Another advantage, more obvious to companies with many data flows: standardized development. The ETL layer uses standard logic for receiving and sending data.

Developing new connectors essentially comes down to reproducing already-built logic, adjusted for a specific data type or the specifics of a system.

Implemented a scalable service bus for ATIMO→

Enabled Polaris to easily launch new products on marketplaces→

Developed a target system integration architecture for a manufacturing enterprise.

Implemented ESB technology and launched 48 data flows→