From spaghetti to flexible architecture in 5 steps: stages of implementing an ESB layer

When a company's IT architecture is just beginning to take shape, it is difficult to assess its future scale. Therefore, by the time that instead of two 1C systems for accounting and warehouse management, it already includes 30-40 systems, the IT team has to work with the historical architecture of the “star” format, which is also, and it is no coincidence, called “spaghetti”.

In this article, KT.team experts explain what steps the company will have to go through in order to replace direct point-to-point integrations with integrations through the ESB layer.

ESB for transparent and easy to maintain integrations

The concept of a service bus (ESB layer) is not new and is not so rare in IT practice.

The ESB layer is a set of IT systems and tools whose sole purpose is the correct, timely and uninterrupted transfer of information between systems.

In practice, the mechanism of operation of a service tire is as follows:

• by integrating, ESB takes information from the source system at a specified frequency;

• transfers this information to DWH (data warehouse) or a message broker that has delegated storage functions (as a rule, we at KT.team have accepted that each type of data must have its own storage — this weakens the links and mutual dependence between systems and services: for example, in the event of an error in one repository, not the entire IT circuit will be affected, but only a minimal part of it associated with this storage);

‍

• retrieves data from storage, usually at the request of the recipient system, but other conditions can be set;

• transforms the extracted data, if necessary;

• enriches (again, if necessary) the extracted data with information from other repositories;

• transmits information to the receiving system in a form, volume and format that suits it.

In fact, ESB takes on the role of the same direct integrations. But point-to-point integrations mean that CRM can share the same customer information 10, 20, 40 times — if this is provided for, respectively, in 10, 20 or 40 integrations. ESB takes information once and distributes it to recipient systems as many times as needed.

The same scheme we showed at the beginning of the article with ESB looks like this.

One of the advantages of modern ESB systems is their simplicity at all stages. A special graphical interface created in the low-code paradigm is provided for developing integrations. It can be mastered not only by a developer, but also by a fairly advanced user, or a citizen developer. Monitoring and logging tools make it possible to detect and localize errors in a timely manner, which simplifies and reduces the cost of support.

At first glance, it is quite attractive. But colleagues — IT specialists will object that in order to implement ESB, the entire architecture will have to be redesigned. We will discuss how to approach this, what to do at each stage to avoid critical problems, and what should be the result of each stage below.

Five stages of developing and implementing an ESB layer

When we talk about the five stages, we are a little “silent”. Figure five refers directly to the number of stages in implementing the ESB layer. But they are preceded by an important and complex stage of solution design and integrations.

At this “zero” stage, you will have to describe what systems are currently included in the architecture, what are their interrelationships and interdependence. It is necessary to highlight the entities and flows of data exchange that already exist.

At this point, you risk falling into a terminology trap. For example, for different systems and different processes, even the simple term “nomenclature” can have different meanings: article plus name; the content of the product card on the marketplace; inventory balances... Therefore, you will have to delve into not only the name of entities, but also their content, look for intersections, sources, conflicts and, most likely, rebuild information routes.

You can deal with this stage either on your own or by inviting an IT consultant to get an “outside view”. But if you choose the second option, we would recommend contacting consultants with experience in successfully building working architectures with an ESB layer. Then the consultant's conclusions and suggestions will be more practical.

1. Selecting an ESB storage location

We suggest starting from this stage before choosing the system itself. Why is that? Different products work in different paradigms. For example, n8n is only suitable if you agree to work via the cloud. And WSO2 can also be installed on the company's dedicated servers. And if, for whatever reason, it's crucial for you to have an ESB layer on your own servers (for example, these are industry security requirements), n8n is not right for you by definition.

If your storage location isn't dictated by industry standards or your own internal requirements, answer these four questions for yourself.

1. Do you plan to scale the company or IT architecture multiple times in the next three years? Your own server gives you more freedom in this matter.
2. Is it important for you to set up ESB as quickly as possible? The cloud version has a higher deployment speed and settings.
3. How high are data security requirements? The server is better protected from attacks and losses.
4. Is storage costs an important limitation? Own servers are a more expensive option.

2. Choosing an environment for developing integrations

There are dozens of systems on the market where integrations can be developed. Almost all of them have a graphical interface and are built in the low- or no-code paradigm. But each such system has its own functional features that allow it to be included in the pool of suitable or inappropriate systems.

For example, Talend does not have the ability to create a reusable routine from a piece of integration in the list of functions. If your architecture assumes that an action is repeated in many integrations (for example, converting data from format No. 1 to format No. 2), it is better to choose a system with the ability to create and reuse subroutines.

If you have products from the Salesforce ecosystem in your architecture, it makes sense to consider Mule ESB. This system is also part of the Salesforce software line, so it will be a little easier to integrate it with other products.

If you have a restriction on using only domestic software, the above products will not work for you. It is better to pay attention to the Russian Datareon.

Don't forget about the financial model of products either. For example, Mule's integration platform itself is an open-source solution and is provided free of charge, while connectors for working with systems are available only on a paid basis. And WSO2 offers the purchase of an annual license.

We will tell you more about the cost of integrating and owning software in another article that will appear on the blog.

3. Selecting a component stack

In addition to an environment for developing integrations, the ESB layer should include monitoring and logging systems, as well as data storage.

Many ESB systems have built-in monitoring and logging tools, as well as a prescribed list of preferred (compatible) products. Therefore, it makes sense to select components only after you have decided on a development environment.

Sometimes we find that at the stage of choosing a stack and implementing ESB, the customer suggests skipping monitoring and logging in order, for example, to reduce the cost of the project or accelerate the emergence of an MVP.

However, KT.team experts do not recommend resorting to such solutions, as monitoring and logging systems help identify errors in time and localize them. If something goes wrong (which, as you know, happens even in the most sophisticated architecture), you can precisely localize the problem, fix it, and prevent it from happening again.

4. Setting up and implementing an ESB layer

At the “zero” stage, you analyzed what data flows looked like in the old IT architecture (as is model), but now your team will have to set up a new architecture (model to be).

What is important to consider at this stage? First of all, don't fall into the trap of terminology and familiar processes again. Attempts to thoroughly reproduce everything and everything in the new architecture are dangerous: this way it will inherit not only successful solutions, but also accumulated errors, conflicts, and desynchronization in processes.

Second: don't try to adjust the tool to familiar paradigms. Even in practice, in a low-code tool, developers still wrote integrations in code instead of using graphical tools. Therefore, experience with low-code (or better, experience in building integrations correctly) will be a plus for the implementation team.

Since there are not so many specialists in this field, and hiring people for a short-term project is irrational, it makes sense to contact an integrator. This will at least save you time, which under other circumstances would have been spent on getting to know the system and training the team, and will help you avoid “rookie mistakes”.

The implementation phase itself involves:

• deploying the integration platform and other ESB layer components on the company's servers (if this option is selected) or connecting the integration platform and other components to your architecture;

• planning integration scenarios for systems (what the ESB layer should take from each system, where to put the information, what triggers it should use, how to transform and enrich this information in each scenario, to which system and with what frequency to transfer it);

• implementing integrations according to the scenarios described at the previous stage;

• debugging and debugging.

If the pre-project work on the implementation of ESB was organized and carried out correctly, the implementation phase itself will not take much time. In our experience, it takes from one to three months to start working in the new architecture at least in terms of processes. Otherwise, implementation may take six months, a year or even longer, and this time will be spent mostly redesigning integrations.

5. Documenting

This stage partly takes place in parallel with the implementation and configuration of ESB. The implementation team prepares detailed documentation on building integrations: where to go to perform the required action, what blocks, elements and subprograms to use, how to check the performance of integrations, etc.

This manual will allow you to set up new integrations and support old ones autonomously and independently of the implementation team, including through the efforts of experienced users who are not developers.

How long will it take to develop?

According to KT.team's experience, on average, it takes about a month and a half to develop and implement an ESB layer before the MVP is launched. We do not take into account the analytical stage here, since its duration depends on the complexity of the existing and future architecture.

‍