When to use a data or event broker in IT architecture: which scenarios help increase the flexibility and reliability of business systems
Published on: 6.8.2024 Reading time: 6 min. Some sources claim that a message broker is always part of an ESB (enterprise service bus). Some even believe the broker is the core of an ESB or that the two terms are the same. In this article, we will examine whether a bus without a broker is possible and in which cases this approach is better for IT architecture.
Let us turn, as we have done many times before, to a simple and clear example with an online store.
Imagine a large company selling auto parts through an online store.
You can roughly model which systems are part of its environment: a PIM system that stores product information; a WMS system for warehouse management; the online store itself; a CRM system that stores customer and discount data; and an OMS system for order management.
They exchange information with each other through the ESB layer at a certain frequency. For example, the online store sends orders with statuses once a minute or once a second.
Of course it is needed. And the broker will send the entire message queue from the online store to the OMS.
But our company is large and sells across all of CIS, not only retail but also in small wholesale.
That is why its landscape includes several OMS instances to manage orders from different regions transparently: the Far East and
Kaliningrad Oblast, for example, as well as wholesale and retail orders.
Has the architecture become more complex? Then the logic has become more complex too. One OMS needs only orders from
Central CIS; another needs only small wholesale orders for Siberia; a third needs only orders in a specific status; a fourth needs only tire orders with the previous status "paid" and an additional flag that the ETL connector must set during processing... And the broker still sends each OMS the full queue of new, canceled, and unconfirmed orders from the online store once a minute. That can be as few as 3 records or as many as 10,000. What happens?
Each OMS system needs to receive all messages for all orders, read them, and determine whether it needs that specific order or whether it has the wrong status or the wrong region.
This is extra logic that has to be coded inside the system, making it heavier and slower.
In this case, one ETL connector pulls all order statuses from the online store and stores them in a database. All the logic, regardless of how complex the conditions are, lives in the connector layer: they sort and pass on only the records needed by their OMS systems. Extra information does not overload the receiving systems, and in fact never even reaches the connector.
Another case is real, from one of our clients.
The company sells household chemicals, dietary supplements, and health products both online and offline.
Some orders are placed through the online store, others through salespeople or field representatives in branches.
The loyalty program for offline customers is tied to a combination of full name and contact details.
But situations like this happened: after clicking Confirm, the customer might remember that their phone number had changed since the last visit.
An order had to be reissued with the same product list, the same full name of the buyer, but a new phone number. In the end, the broker passed both orders to the CRM system, and duplicates had to be removed from each system manually. The case we described is not the only similar one we know of.
When using a broker, duplicate messages reach downstream systems quite often. To get rid of them, you either have to clean up duplicates manually, or add extra logic in the receiving system for example, the system must find duplicates by key parameters, compare the send time, remove earlier records, and keep only the newer messages.
If you use a database plus an ETL layer in the ESB environment, you can move message reconciliation logic there. In that case, only clean information will reach the receiving system, and you will not need to use additional system resources to process duplicates.
Now imagine that you decide to connect a new system to the stack, such as BI, to track metrics and analyze hypotheses.
You get one more system that starts listening to message queues for product cards, views, orders, and so on.
That is enough to get fresh data.
But for some conclusions, that is not enough: retrospective data is also needed. That means you will have to request a data re-export from the source systems.
It would seem that a broker should let you get this export as quickly as possible. Yes, but no. First, if your source system has many duplicate messages, that will definitely slow down export and processing. Second, keep in mind the asynchronous nature of the workflow: delays of 10-15 seconds are perfectly acceptable even under heavy load and in very large environments. It is impossible to predict how many records will be missing from the export in such a setup.
If your bus uses databases instead of a message broker, you already have a ready-made, deduplicated data export from all the required systems. You have orders, product cards that can be matched to orders, delivery statuses, and everything else that may be needed for business analytics (as in our example) or any other new system or process. There is no need to rebuild anything from scratch, re-export data from source systems, remove duplicates, and so on.
Sometimes you need to make a complex selection for external consumers, and in that case a broker is almost useless.
Your PIM system stores product cards:
They are not for sale because you are only buying them in, or they are seasonal products, or there are no deliveries yet there are many possible reasons.
But the marketplace needs to receive from you: items that are in stock, marked as "in stock"; those that will arrive soon, also with the appropriate label; and those that will soon run out and that you do not plan to restock, marked as "soon out of stock".
To build such data flows, you need to read several sources, compare them, merge them, and only then send the result.
To implement this logic with a broker, you would have to hard-code export restrictions for the marketplace on the PIM system, WMS system, procurement management system, and possibly a couple of other sales-related systems.
With a large assortment and a complex catalog, you will either have to schedule exports for off-peak hours or accept slowdowns that affect key parts of your IT architecture.
When using a database plus ETL, all logic moves to the connector layer without slowing down any of the source systems.
It seems that a broker may be a good fit for processes with no complex logic where maximum transfer speed matters.
For example, in one of our projects, speed was a key metric: there were not many records, but their status was constantly changing.
We ensured continuous status updates with a broker, but the receiving systems simply could not keep up with the incoming stream of update messages.
Paradoxically, the updates, the things that changed most often, were exactly what the consuming systems did not need.
They were interested only in the records themselves.
As a result, batch processing from a relational database provided the best speed: it did not overload the receiving systems and delivered all the required data to them.
KT.Team has dozens of integration cases with different business logic. Across our projects, we have seen different logic, requirements, and data processing issues. We know when it is better to recommend a broker and when a database is the better choice. If you have doubts, book a consultation and our experts will help you choose the right tool and design integrations correctly.
