How to Choose the Right Type of Integration for Your IT Architecture

point-to-point, broker, ESB 24.4.2024

We examine the specifics, pros and cons of three integration types: point-to-point, via a broker, and via an ESB.

Reading time: 12 min. The internet offers plenty of articles about point-to-point integrations, brokers, and ESB.

As a rule, they focus either on the technical details of building integrations or on the obvious pros and cons of each approach. In this article, KT.Team experts look at all three integration types from the perspective of their internal structure — this highlights not only the obvious but also the hidden advantages and drawbacks of each.

Prefer to explore this material as a podcast?

Regardless of the integration type, every integration includes: a source system; a system that receives the information; an ETL layer (extract, transform, load); and a data warehouse — the space where information transferred through integrations is stored.

This is a very high-level view that ignores the nuances and logic of each specific integration.

But it is precisely this basic structure that will help us understand the pros and cons of each integration type.

After all, an integration's properties depend precisely on where the ETL functions and the data warehouse sit relative to the systems.

The first and biggest drawback of this integration is strong mutual dependency between systems.

Let's continue our example with 1C and an online store.

For a user to place an order, the online store must receive the user's loyalty program status and address from 1C.

If 1C goes down or slows under heavy load, critical information stops reaching the online store.

Customers will not be able to place orders until 1C is back up.

To protect the business from such risks, the development team will keep inventing safeguards: setting flags, devising queues, reinventing asynchronous messaging between systems. In the end, the "inexpensive" direct integration will catch up with and overtake the cost of the "expensive" bus.

Drawback number two: systems start to "know" too much about each other. In what format and at what rate should data be delivered? Which data is redundant or insufficient?

All of this has to be accounted for in the integration logic.

Third drawback: integrations overload the source system, even if it initially works with a small amount of data. For example, a CRM holds customer data that is constantly accessed by the online store, delivery, marketing applications, and so on.

As a result, the load on the CRM grows many times over due to constant requests from other systems.

Fourth drawback: a change in any system triggers a cascade of rework across all related integrations and systems.

Add a new attribute to a product card — and you have to change the logic in every system that consumes that card. And the logic change does not always go smoothly: in especially critical cases there is nothing left to do but roll all the changes back.

The fifth drawback is proxying. For example, both 1C and the online store need to receive stock information from the warehouse software.

To avoid overloading the warehouse software and transferring the same data several times, developers may offer you an "elegant solution": let the WMS send everything only to 1C. And 1C, in turn, passes to the online store not only its own portion of product and order data, but also the stock levels.

The accuracy of the stock data shown in the online store now depends on two systems working at once. 1C (or any intermediary system) ends up overloaded with data outside its core purpose.

In practice we have seen even longer data enrichment chains — they are not that rare. And if something goes wrong in the target system, you have to run an investigation: where the data transfer failed, why, and how to prevent it from happening again. The last and, in our view, most critical drawback: point-to-point integrations block or significantly slow down IT system upgrades.

If in three years you decide to replace that same CRM, you will have to redevelop and reproduce the logic of dozens of integrations — and not only the ones directly connected to the CRM.

Meanwhile, the company becomes hostage to the developers who know a lot about the existing logic.

Giving them up or switching the team to other tasks becomes almost impossible.

The developers themselves become hostages of their own knowledge: even on vacation they must always be reachable by phone — because who knows what might go wrong?

In this type of integration, besides the source system and the receiving system, an intermediate layer appears — a message broker.

Systems do not interact with each other — only with Apache Kafka, RabbitMQ, or any similar service.

At the same time, IT systems still initiate the sending and receiving of information. In this integration type, information storage is moved outside the end systems: once information has reached the broker, the receiving system picks it up from there at the agreed time.

At the same time, the ETL logic still stays on the side of the systems taking part in the integration.

At different stages of data transfer, responsibility for extraction, transformation, and loading is split between the source system and the receiving system.

Data exchange becomes more controllable.

In the broker you can see which messages were sent out and which were delivered, check the logs, and review the event history. Second: source systems no longer need to find out whether the receiving system is available at the moment a message is sent. Their area of responsibility is limited to the logic of forming their own message queue.

Another advantage of brokers worth highlighting is contract support.

A contract is a check of a message against basic rules (the presence of a specific field in the required format). Equally important in this approach, development becomes more standardized.

Almost every service has tools for connecting a broker. This means no development from scratch, no workarounds, and no need to reinvent integration logic every time.

It is also worth noting that data transfer through a broker is fast — here the broker is almost on par with point-to-point, the fastest integration type.

The first major drawback: using a broker puts load on the service that sends the message.

Imagine a situation: the broker is unavailable for some time, and no new messages are added to the queue during that period.

The broker itself does not "know" what any given system tried to send to it. So the source system has to check which message was the last one in the queue and resend everything that accumulated during the downtime.

This logic puts extra load on the system, so there is a temptation to drop it.

But in that case, undelivered messages will sooner or later affect critical business processes.

The second drawback is that the receiving system gets overloaded with unnecessary data.

A broker cannot choose which messages to pass on — it delivers the entire queue. For example, all assortment information or all warehouse stock information from the WMS.

Now imagine that your system does not need the full volume of certain data. For example, the 1C instance running in your Ufa branch does not need customer and order data from all 20 of your branches.

But a system cannot "ask" the broker for filtered data.

It has to receive through integrations everything that the CRM and OMS (Order Management System) send, store it in its own databases, and use internal logic to separate the 5% of needed data from the 95% of excess data.

The third significant drawback is that the contract gets set in stone.

All participants in the process must agree on the format in which messages will be transmitted.

This removes flexibility and increases the labor costs of maintaining a single format.

A less obvious drawback is the lack of a convenient and/or detailed retrospective. For example, the Kafka broker can store historical data — but if something breaks, you have to reprocess the entire message queue from the historically first record to the last.

It is impossible to run any queries on current statuses.

The problem is most acute when you need to share history with another team or perform a migration — extracting this information will be problematic.

This drawback leads to another significant downside: the impossibility of reliable reconciliation.

Systems load both old and new messages from the broker without distinguishing between them.

During reprocessing, a newer message may end up overwritten: for example, an order gets the status "created" instead of "delivered".

Such situations can be avoided, but that requires adding an intermediate reconciliation service or building reconciliation logic into the receiving system.

This also affects speed. On the one hand, systems quickly exchange the entire message stream, but on the other, figuring out what is current and eliminating duplicates is difficult.

That is why reading data is much slower.

Using a broker can lead to proxying just as easily as direct integrations can.

To keep every system in the landscape from receiving redundant data, it is "easier" to route it through one system, normalize it there, and pass the normalized data on. We described above where this leads.

There is also a hidden drawback of a managerial nature. Every system has its own product owner.

But who owns the broker and the integrations tied to it?

Most often, it is the technical department that takes on this role.

For the other departments this is a clear plus (less responsibility), but in that case none of the stakeholder departments will feel in control of the infrastructure.

The basic structure of ESB integrations helps explain where their advantages come from.

This is the only integration approach where all three components are separated and each performs strictly its own function: systems generate and consume data, the ETL layer delivers it to the right destination in the right form, and the storage (logically enough) stores the data.

One result of this is, for example, reduced load on services, since they contain no integration logic.

The service's job is to provide access to its data via API (for a modern service), FTP, or any other exchange method the service supports. The ETL layer itself will pick up the data and pass it on to the warehouse.

The service "thinks" only about keeping all the information it is responsible for up to date and complete.

The second advantage follows from the first — the source system does not need to know the technical requirements of the system consuming the data.

Data format, transfer speed, update frequency — all this logic lives in the ETL layer.

The load on each system becomes controllable. In the ETL layer you can regulate the system load and change settings: for example, keep 1C free of load during the day and update data only in the evening.

Moreover, you can control the request rate: how many packets per unit of time are delivered to the service.

The bus takes on information filtering. In the previous section, we looked at the example of a branch's 1C that has to pull 20 times more information than it needs to operate. With integration via an ESB, this does not happen.

The bus sends a service only the data it really needs — all it takes is building that logic into the connector.

A properly built ESB integration implies fault tolerance of the entire IT architecture.

Services do not exchange data directly: the ETL layer saves it to dedicated storage and retrieves the current version from there when needed. For example, if your bonus management system is temporarily unavailable, it will not affect order placement in the online store in any way.

The online store will get the bonus points and personal discount data from the data warehouse.

When the bonus management system comes back online, the connector will inform it about redeemed points and new orders so it can recalculate personal discounts for repeat customers. In a broader sense, using an ESB provides flexibility and resilience to cascading changes. Imagine you decided to replace a foreign CRM with a CIS one.

If you chose the point-to-point integration type, you would have to, first, redevelop the point-to-point integrations between the CRM and other systems, and second, modify the integrations of other systems that use CRM data. With an ESB, you only need to develop the flows out of and into the CRM.

The flows of consumer systems remain unchanged.

It may seem that using an intermediate layer reduces speed.

In practice, however, the exchange speed is very high: tens, hundreds, thousands of messages are delivered within 400-500 milliseconds! Another advantage, more obvious to companies with many data flows: standardized development. The ETL layer uses standard logic for receiving and sending data.

Developing new connectors essentially comes down to reproducing already-built logic, adjusted for a specific data type or the specifics of a system.

The main drawback is the upfront effort.

To do integrations right, you need to stop and think about the IT architecture: define each system's areas of responsibility; understand what information it receives and from where, what it sends, which of it is necessary, and which is redundant.

Here it is important not to overcomplicate the overall approach and to think strategically, so that the chosen model proves viable for years to come.

The effort required to prepare the bus for launch also shows in the seemingly high cost of the integrations.

Right from the start you will have to pay for business analysis, for designing the integration logic, and for implementing the bus tools — the ETL layer, DWH, and so on.

In our experience, this is about 90% of the cost of implementing and owning an ESB.

The remaining 10% is support for integrations already in place.

Misunderstanding the approach or lacking experience with ESB means the person responsible for implementation may build the integration incorrectly — for example, by pushing excess business logic into the intermediate layer.

This affects both immediate development speed and the cost of future enhancements.

The worst outcome of poorly designed bus integrations is turning a flexible, transferable architecture into a monolith.

Instead of keeping things simple and transparent, the team writes an enormous number of lines of code.

Instead of loose coupling and fault tolerance, the result is an architecture that is hard to maintain.

To keep things from getting to that point, you need a competent implementation team: the entry threshold here is higher than for integrations

Implemented a scalable service bus for ATIMO→

Enabled Polaris to easily launch new products on marketplaces→

Developed a target system integration architecture for a manufacturing enterprise.

Implemented ESB technology and launched 48 data flows→