Point-to-Point, Broker, ESB: Which Integrations Help Build a Loosely Coupled IT Architecture

25.7.2024 In this article, we look at which types of integration help build a loosely coupled IT architecture and which lead to a monolith.

We evaluate each integration type, point-to-point, broker (Kafka), and enterprise service bus, across five parameters. Reading time: 11 min.

Loose coupling is the foundation of a flexible IT infrastructure.

At the same time, the mere presence of integrations in a landscape is not an indicator of a monolith or its absence.

What matters is how these integrations are implemented.

One approach can lead to a tangle of hard-to-maintain connections and, as a result, difficult business scaling.

We already touched on integration approaches in one of our earlier articles→; today we will look at this topic specifically from the perspective of building a loosely coupled architecture and business flexibility.

No intermediaries, extra storage, or shared context.

The fastest integration type in terms of development: off-the-shelf APIs and minor customizations are enough.

The development teams for different services agree on how they will be used.

This kind of integration is set up quickly, especially when there are only a few services in the environment.

Building a direct integration with simple logic is an inexpensive task.

Message brokers usually support contracts, meaning they validate messages against basic rules, such as the presence of a specific field in the required format.

This kind of integration makes development more standardized.

A broker, like point-to-point, supports high data transfer speeds between services.

The broker burdens the source system with unnecessary actions.

If the broker was unavailable for some time, no new messages were added to the queue during that period.

The broker burdens the receiving system with unnecessary data.

A broker cannot choose which messages to pass on; it delivers the entire queue. For example, all assortment information or WMS data on warehouse stock, even if most of it is not needed by the receiving system.

But the latter still stores them in its own databases and, through internal logic, separates the 5% of required data from the 95% that is unnecessary.

Using a broker leads to contract hardening.

All participants in the process must agree on the format in which messages will be transmitted.

This removes flexibility and increases the labor costs of maintaining a single format.

It is impossible to run any queries on current statuses.

The problem is most acute when you need to share history with another team or perform a migration.

For other departments, this is a plus (less responsibility), but in the end none of the interested departments feel they have control over the infrastructure.

Ease and speed of modifying integrations/services: Partially complies.

Data exchange is standardized, but the need to maintain a single data format can limit flexibility.

Handles peak loads and works without delays: Partially complies.

The integration supports high data transfer speeds, but the source system may become overloaded due to excessive actions and data.

Launching a new data flow without modifying services: Does not comply.

A new data flow requires validation and possibly changes to comply with the broker's contracts.

No problems in other services when one fails:

Ease of monitoring and ability to delegate to support: Partially complies.

The broker allows data and logs to be monitored, but the complexity of retrospective analysis and event detail can make full delegation to support difficult. Total: 2 points of loose coupling out of

The load on each system becomes manageable.

The service's job is to provide access to the data via API (if it is a modern service), FTP, or any other available method. The ETL layer will fetch the data itself and pass it on to the store.

The service only cares that all information it is responsible for is current and complete at the moment of transfer.

Any system can be connected to the environment quickly, and enhancements can be made in any order.

This aspect is the main condition for effective business scalability.

Here is an example of how we solved a scalability problem by introducing an ESB layer at a company that manufactures electrical equipment and comprehensive energy-efficient solutions.

Integrations are standardized, and service logic becomes simpler. The ETL layer uses standard logic for receiving and sending data.

Developing new connectors comes down to reproducing already built logic with adjustments for a specific data type or system-specific characteristics.

Development and monitoring can be delegated. This advantage follows from the previous one.

Development becomes simpler, and monitoring responsibilities can be taken off developers and delegated to support.

A properly built ESB integration implies fault tolerance of the entire IT architecture.

Services do not exchange data directly: the ETL layer stores it in dedicated repositories and retrieves the current version from there when needed.

A fairly high upfront cost.

Right from the start, you have to pay for business analysis, designing the integration logic, and implementing bus tools such as the ETL layer, DWH, and so on.

Here we broke down in detail the costs of implementing the ESB layer at each stage.

Ease and speed of updating integrations/services: Meets the benchmark.

Services can be easily enhanced or new ones connected thanks to the separation of the ETL layer and the data store.

Launching a new data stream without modifying services: Yes.

A new data flow can be launched without modifying services, since the data transfer logic is managed in the ETL layer.

No problems in other services when one fails: Meets the benchmark.

An error in one service does not affect others thanks to data isolation in the ETL layer and storage.

Ease of monitoring and the ability to delegate to support: Meets the benchmark.

Integration monitoring can be delegated to support, since ESB provides the necessary tools for observation and management. Total: 5 points of loose coupling out of