From Spaghetti to Flexible Architecture in 5 Steps: Stages of Implementing an ESB Layer

26.4.2023 We explain the stages a company needs to go through to replace direct point-to-point integrations with integrations through the ESB layer. ESB for transparent, easy-to-maintain integrations

Five Stages of ESB Layer Development and Implementation

How much development time will be required? Article published on Medium Reading time: 7 min.

When a company's IT architecture is only beginning to take shape, it is difficult to estimate its future scale.

So by the time it grows from two 1C systems for accounting and warehouse management to 30 or 40 systems, the IT team has to work with a historically formed star-shaped architecture that is also, and not by accident, called spaghetti. In this article, KT.Team experts explain the stages a company needs to go through to replace direct point-to-point integrations with integrations through the ESB layer.

The service bus concept, or ESB layer, is not new and is not uncommon in IT practice. An ESB layer is a set of IT systems and tools whose sole purpose is the correct, timely, and uninterrupted transfer of information between systems.

In effect, the ESB takes over the role of those same direct integrations.

But point-to-point integrations mean that CRM can provide the same customer information 10, 20, or 40 times, if that is required in 10, 20, or 40 integrations respectively. ESB retrieves the information once and distributes it to receiving systems as many times as needed.

The same diagram we showed at the beginning of the article looks like this with ESB.

One of the advantages of modern ESB systems is simplicity at every stage.

A dedicated visual interface for building integrations is provided, designed around the low-code paradigm.

It can be mastered not only by a developer, but also by a fairly advanced user, or a citizen developer.

Monitoring and logging tools make it possible to spot and isolate errors in time, which simplifies and lowers support costs.

At first glance, it looks quite appealing.

But IT colleagues will argue that implementing ESB will require reworking the entire architecture.

How to approach this, what to do at each stage to avoid critical issues, and what each stage should deliver is covered below.

When we talk about five stages, we are omitting a few things.

The number five refers directly to the number of stages in ESB layer implementation.

But they are preceded by an important and substantial stage of solution and integration design.

At this zero stage, you need to describe which systems are currently part of the architecture and what their relationships and interdependencies are.

So you will have to dig not only into the names of entities but also into their content, look for overlaps, sources, and conflicts, and most likely redesign information routes.

You can handle this stage on your own or bring in an IT consultant to get an outside perspective.

But if you choose the second path, we would recommend working with consultants who have experience successfully building working architectures with an ESB layer.

Then the consultant's conclusions and recommendations will be more practical.

We suggest starting with this stage before choosing the system itself. Why?

If the hosting location is not dictated by industry standards or your internal requirements, answer the following four questions for yourself.

Is a multiple-fold scaling of the company or IT architecture planned over the next three years?

A dedicated server gives you more freedom in this regard.

Is the fastest possible ESB setup important to you? A cloud-based option offers faster deployment and configuration.

How high are the data security requirements?

A server is better protected against attacks and data loss.

Is storage cost an important constraint?

Owning your own servers is a more expensive option.

There are dozens of systems on the market that can be used to build integrations.

Almost all of them have a visual interface and are built on a low-code or no-code paradigm.

But each such system has its own functional characteristics that determine whether it belongs in the pool of suitable or unsuitable systems. For example, Talend does not include the ability to create a reusable subprogram from a piece of integration.

If your architecture assumes that a certain action repeats across many integrations, such as transforming data from format No. 1 to format No. 2, it is better to choose a system that supports creating and reusing subprograms.

If your architecture includes products from the Salesforce ecosystem, it makes sense to consider Mule ESB.

This system is also part of the Salesforce software lineup, so integrating it with other products will be somewhat easier.

If you are restricted to using only domestic software, the products listed above will not be suitable.

It is better to consider the CIS Datareon.

We will cover the cost of integration and software ownership in more detail in another article that will appear in the blog.

At the zero stage, you examined how data flows looked in the old IT architecture (the as-is model), and now your team needs to configure the new architecture (the to-be model).

Attempts to reproduce everything in full detail in the new architecture are dangerous, because it will inherit not only the successful decisions but also accumulated mistakes, conflicts, and process misalignment. Second, do not try to force the tool to fit familiar paradigms.

Even in practice with a low-code tool, developers still wrote integrations in code instead of using the visual environment.

Therefore, experience with low-code, or better yet, experience in building integrations correctly, will be an advantage for the implementation team.

Since there are not many specialists in this field, and hiring someone in-house for a short-term project is not rational, it makes sense to turn to an integrator.

At a minimum, this will save you the time that would otherwise be spent getting familiar with the system and training the team, and it will help avoid beginner mistakes.

If the pre-project work for ESB implementation was organized and carried out properly, the implementation stage itself will not take much time.

In our experience, one to three months is enough to begin working in the new architecture for at least some processes. Otherwise, implementation can drag on for six months, a year, or even longer, and most of that time will be spent reworking integrations.