We explain why ESB is often criticized, what its strengths and weaknesses are, and when a data bus is truly useful.
10.2.2022 ESB is a specialized integration development tool that has been evolving for more than 20 years. Yet many developers still speak against it. Let us look at the main objections and understand why they arise.
The system that knows Integration with middleware: do not reinvent the wheel Why do developers oppose data buses? "Our company does not have the required technical stack"
"The bus is not real" "ESB is outdated" "I do not believe in low-code" "Had a bad experience or heard about one" Overcoming resistance Reading time: 9 min. Imagine you want to start selling through Wildberries. You have already built the sales chain, warehouses, and store network - the business is operationally running.
You approach your 1C developers with this request: "We have already signed the contract, we need to ship to Wildberries, and we need an integration."
The task seems simple and clear... Let us look at what hidden pitfalls the "traditional way" has
solutions to tasks like how to simplify integrations with middleware and why technical people often oppose the use of data buses.
So, your developers took the task and estimated it: "We will do it next month."
Or they estimate that they will do it tomorrow, but in fact a month still passes before the project is delivered. The deadlines are driving you crazy, but there is nothing you can do.
After a month of work and another half month of debugging, you think you will finally start selling your products on Wildberries.
After all, it seems all products are published and nothing is blocking sales.
But a week later it turns out that some products on the marketplace are not updating, some are missing altogether, and some have incorrect stock balances.
A small note: here we are not talking about cases where you process orders yourself - there you would lose a couple of orders as well, which would lead to a drop in marketplace ratings and so on and so on.
"The task is complex, we are integrating with Wildberries for the first time. (Or not for the first time, but it used to be better!)"
- Resources are limited. - If only it were not four people, but forty, then wow! Replace Wildberries and 1C in this example with any other systems, for example SAP and electronic document management, or CRPT, or anything else - the scenario will be roughly the same every time. What do you see? You promised to launch sales this season.
The development team says you are putting a lot of pressure on them and that it is impossible to work. "We have been overworking. Give us a bonus!"
Or maybe they will just quit: in a hot job market, developers will find work. And you will be left with a project that never launched, barely works, and nobody knows who will support it.
Why does the simple task of making a fast and reliable system integration in our hyper-automated era turn into a catastrophe?
For nearly 20 years, there have been tools on the market that let teams manage integrations between systems quickly, reliably, and transparently for everyone.
Moreover, such tools are a foundational concept of service-oriented architecture.
Why many teams do not use this approach or use it incorrectly will be discussed at the end. Let us start with solving the task.
Collectively, such tools are called middleware, that is, an intermediate layer.
This layer by itself does not seem to provide value at first glance.
But a smooth road between your home and the restaurant also does not bring direct value.
that is, a software package that lets you take data from one system (for example, products from 1C), store it in intermediate storage, and send it to another - for example, Wildberries.
This layer "knows" that products cannot be transferred directly from 1C to Wildberries, and that the "color" attribute, for example, needs to be transformed in a special way.
This layer can also "know" that 1C does not store all product information, and that product photos must be taken from another system of yours, or even from the internet, before sending them to the marketplace.
What provides speed, reliability, and transparency?
Speed comes from the fact that your 1C developer does not need to write logs, set up monitoring, or learn how Wildberries works, and so on. The 1C developer already knows that if their system is the master system for products, then their responsibility already includes the bus connector and, in fact, the quality data in 1C. In other words, they can set up the transfer long before the export task even appears.
For example, your 1C may be responsible for products and prices, while the warehouse system is responsible for stock balances.
It is also responsible for creating and updating this data in the intermediate layer - for storage in the corresponding broker queue.
When an integration task with a marketplace arrives, the technical team responsible for sales on the platform analyzes the Wildberries API and aligns the fields.
You may have only one developer responsible for everything, including assembling the integration in the bus interface.
But in a bus, in most cases, they will not have to develop anything.
It is enough to use the ready-made integration builders in a tool that has been developed for many years specifically for this task - that is first.
ESBs provide a huge toolkit for all common operations, and often include prebuilt connectors for popular services. A developer using an ESB spends less time on code and more time designing the integration.
Second: the developer does not have to reinvent the wheel
for system monitoring and action logging.
Native support for these procedures in buses allows the developer to understand more easily and quickly that there is a problem somewhere.
Because the developer spends less time on mechanical work, they have more time to debug and validate systems. Third, changes in connectors to end systems happen faster than changes in the development itself.
According to our statistics, if small integration fixes take four hours or more in code, then in the bus builder it takes 15 minutes. On average, when creating integrations with buses, developer effort is reduced fivefold.
In addition, sometimes your 1C team is overloaded, but you will not be able to bring in a technical specialist who is completely unfamiliar with your 1C for this task.
With buses, provided a service-oriented approach is followed, you can scale the team without difficulty.
1C does not have special connectors for Wildberries, but they are not needed because the bus uses the standard out-of-the-box API of your 1C. As a result: your 1C developers will not be distracted by integration support; your sales channel will not stall just because a higher-priority implementation task arrived in 1C; there is less code in the end systems.
Do not forget that writing an integration itself is not a particularly difficult or interesting task. So a 1C team freed from that task does not burn out. It seems like all upside.
But usually things are not the way we describe in this article...
We have gone through bus implementations many times, both in our team and in other organizations, and we know the developers' arguments very well. Here are the main ones.
Developers believe that without knowledge of the required technical stack, mainly Java, it will not be possible to work with a data bus. Yes, your company may not have a Java developer.
But a bus already has many ready-made builder components, and creating new ones requires a clear, limited knowledge domain.
The lack of Java developers does not become an obstacle.
Developing from scratch in any language, and especially burdening current systems with integration logic, is, no matter how you look at it, a more labor-intensive path that is also worse from an architectural standpoint.
Developers often say that writing integration in code is more "real": faster, more reliable, and clearer.
Another version is: "I do not want to work in the bus"
because "the bus lacks development tools that are very needed, such as versioning".
The developer will tell you that it takes a long time to figure out the bus, while "with code I can do it right now! It will be ready in a couple of days."
A couple of days usually turns into a month of problems. And yes: learning is physically painful.
Ask yourself whether your team will build in a month an integration better than bus developers have built over 20 years.
How many thousands of integrations have passed through bus tools, and how many problems have been found and solved over all this time?
In our experience, analyzing a developer's excuses always led to the real reason: they simply did not find the tools and assumed there were none.
The idea that ESB is outdated comes from a lack of understanding of the fundamental principles behind service and microservice architectures. We will discuss this separately in one of our future publications.
For now, we will only say that the founder of SOA and author of the twice-republished book "SOA: Analysis and Design of Service and Microservice Architecture"
Thomas Erl does not think so, and neither do other great minds in computer science. We strongly recommend that both your colleagues and the business read this book.
Individual developers may not believe in low-code solutions, but the market, on the contrary, shows trust in this approach.
Just look at the sales of Salesforce, one of the largest low-code solutions in the world.
It is important to explain to the team that with low-code tools, and data buses in particular, their work will become more meaningful and interesting.
Why spend time on problems that someone else has already solved before you, when you can work on something fundamentally new?
Let us be honest: implementing a bus correctly is not an easy task.
To solve it successfully, you need to understand SOA principles, low-code principles, and want to do it well within the tool's paradigm rather than the way you are used to.
We have seen a huge number of incorrectly implemented buses that created more problems than they solved.
Well, to drive a car, you first have to learn how to drive. And the first attempts behind the wheel can also end with negative emotions.
But that does not mean driving is a bad idea in principle, right?
All these objections come down to a lack of understanding of fundamental principles, fear of the new, and fear of responsibility for the result.
When the developer has less room to hide behind technology complexity, when their work is visual and easier to interpret, when they have a clear area of responsibility, when they work with tools that have been refined for two decades solely for integrations, their first reaction will inevitably be self-defense.
They will try to explain why it is better not to implement a data bus and not allow any changes.
But in reality, implementing a bus benefits everyone.
Assembling integrations from components becomes a good entry point into the profession for junior developers.
They start bringing value to the team almost immediately, gradually increasing their level of involvement in the project.
At the same time, it is harder for them to make a mistake.
An experienced developer, after implementing the bus, shifts focus to the most complex tasks: writing new components and deeply analyzing system behavior.
The integration customer gets a self-documenting system whose behavior is understandable, and in some cases can even take part in configuration. Everything we discuss in this article has been tested through our management experience. In our work with our own team, while training our clients' teams, and while consulting and managing our clients' IT infrastructures.
We admit that it is difficult for an unprepared manager to implement buses. You have to overcome resistance, ignorance, and a "work-to-rule" slowdown.
For example, on one of our own projects we did not pay enough attention to onboarding the new developer. As a result, we saw integrations that were de jure in the bus, but de facto were indistinguishable in their properties from the "traditional" integrations we discussed at the very beginning of the article. Without sufficient technical authority in the team, pushing through these changes is even harder, but that is the manager's job - to think through and implement changes.
