Companies approach the implementation of the ESB layer with different sets of tasks and different problems. KT.team customers who are planning to implement ESB most often cite the following reasons:
- adding new systems to the circuit — and therefore scaling up the business — has become long and expensive;
- the more integrations there are, the more time developers spend supporting them, rather than developing IT products;
- there are one or more “points of failure” in the circuit — systems whose failure leads to failures in other systems and databases, even if they are not directly connected, etc.
If you regularly face these problems, you've probably also thought about the integration bus. But to start a transformation project, it is necessary to take into account all the risks and understand how long the transition will take. And won't it be more expensive for the company than the “leave it as it is” strategy.
But as Brian Foote and Joseph Yoder wrote in an article Big ball of mud, “if you think it's expensive to build the right architecture, calculate how much bad architecture costs you.”
In this article, KT.team experts will answer common questions that IT directors face when implementing ESB: will the new architecture be effective for their company's business? What budget is realistic to expect and how can this budget be protected before a CEO or other business customer? How can I track the results of each phase of the project?
ESB bus implementation stages and overall project duration
The project to introduce an ESB bus can be divided into three big stages:
- pre-project stage (analytics, pre-project survey, planning of future architecture);
- integration of systems included in the ESB layer, development of streams and connectors;
- creating documentation for the configurator to work.
It is quite difficult to accurately predict the duration of a project: it depends on the number of systems and flows. On average without taking into account the pre-project period, the introduction of the ESB layer It takes one and a half to two months before the MVP is launched.
Where do case studies about one and a half or two years (at best) come from then? There is no definite answer. Sometimes the launch of an updated MVP architecture is delayed because you need to link dozens or hundreds of systems and build thousands of connectors. But more often this period is due to the fact that the pre-project survey stage has been missed. As a result, the implementation project does not take into account important details regarding the relationship between systems, data sources and flows, middleware bandwidth requirements, etc. The project team has to make changes to the project “on the go”, change priorities, return to ready-made flows and rethink them.
Why is this happening? Most often, the reason is that the company does not have a single knowledge center for all processes and relationships. Each engineer has a deep understanding of their area: a cluster of systems and integrations. But no one has the full picture, because there is no need for it either.
The pre-project stage of implementing the ESB layer allows you to close this blind spot.
Pre-project phase: approximate dates and results
Duration: from 30 working hours to 2-3 weeks (depending on the number of systems and the complexity of the project).
Stage results:
Fixing the requirements for the new architecture
The first task facing the implementation team is to understand what problem the client came with and what system requirements should be covered by the ESB implementation. Implementation alone is never a goal, but only a means to solve problems or business problems.
For example, a startup that automates the receipt of waybills for fleets, contacting KT.team, wanted to simplify future scaling and reduce the number of errors in data transfer. The increase in the number of partners already made the system overloaded in the top ten connected taxi fleets, as direct integration took up a lot of memory on the company's servers.
The lack of a single data format and the potential vulnerability of direct integrations were also a factor in choosing the future architecture.
After analyzing the customer's requests, the KT.team compiled a list of requirements for the new solution:
- a survey of one taxi company should require less memory;
- adding a new counterparty should be quick and easy;
- the counterparty should not have direct access to the client's server and database;
- data should be received every 20 minutes or more.
Based on these requirements, the implementation team was able to work out the configuration of the new architecture and move on to choosing a stack for the project.
Analysis of the existing architecture
Sometimes this step looks optional and can be seen as a waste of money. It's easy to understand a CIO who refuses analytics and asks them to start building a new architecture right away: he is sure (or almost sure) that:
- the team already knows how things work, and
- It is more important to develop a new architecture than to understand the old one.
This is true, but with some caveats. As a rule, the team really knows how systems are interconnected and how existing integrations work. But locally, that is, individual engineers know their area of architecture “to the comma” and have a very vague idea of the rest of its parts. Most likely, there is no single idea anywhere.
Why is this dangerous? There is a high risk of transferring old problems to a new landscape. But the main task of implementing ESB is not to “implement”, but to “improve”.
Therefore, before proceeding directly to the stage of implementing the ESB layer, you need to describe the current architecture, establish all connections between system entities and data exchange routes, identify conflicts and duplications, and record master systems for each type of data.
Dozens of interviews and diagrams developed together with the company's technical specialists and describing the current state of affairs will be required. Based on these schemes, the As-Is map will be formed — a complete picture of the company's IT landscape at the start of the project with all the difficulties, failures, bottlenecks and successful solutions. It will help develop a new architecture that is more flexible and scalable.
The analytics phase is one of the most time-consuming and time-consuming during the pre-project implementation phase. But later, it saves three to five times more time than it takes.
The choice of tools for implementation
Based on the results of the analysis of the task and the existing project architecture, the implementation team proceeds to the stage of selecting tools.
This stage involves working with hypotheses and working closely with the customer to avoid project rollbacks due to an inappropriate tool at the development stage.
A network company specializing in the distribution of eco-products contacted the KT.team to implement the ESB layer. The customer's main requirements were: to unload the source system (CMS), eliminate the problem of duplicate messages in the system, make the system more fault-tolerant and provide protection against errors when delivering messages.
Before starting the project, we conducted analytics and selected tools for the project:
- Kafka message broker
- Mule with the ability to cluster as a bus base
- Elastissearch Logstash Grafana logging system
However, while discussing this configuration with the customer, we had doubts that with this combination we would be able to quickly connect new systems and implement the functionality of re-sending messages over a certain period of time.
When solving the problem, the KT.team hypothesized that to solve the project's problems, it is worth using the MongoDB intermediate data store instead of a message broker. The use of intermediate storage would also allow the customer to later use the BI system for detailed analytics.
Thanks to the changes made, the draft solution became more functional and was able to cover all customer requirements. The final configuration was approved and we moved to the implementation phase.
Building and approving the architecture project for the future solution
The last step of the pre-project stage is the validation of the project architecture within the team and final approval with the customer.
To minimize the risk of project delays, kickbacks and reworks, KT.team has developed a business process to prepare for the implementation of integrations:
- After receiving the customer's requirements and initial analysis of the project, the project manager and developer draw the connector's logic in the form of a BPMN diagram.
- The next step is to brainstorm the team on possible ways to optimize logic: reduce costs, speed up development and simplify implementation.
- The scheme is adjusted taking into account tested hypotheses.
- If the integration is confirmed to work and there are no additional inputs, the team's software architect validates the BPMN scheme.
- The logic of the solution is discussed and agreed with the customer.
- If the architecture project is fully approved, the team starts developing it.
Solution development phase
Average duration: about 40 hours per developer per thread. The overall duration depends on the complexity and number of threads, as well as the number of developers in the implementation team.
The main results of the stage:
- The systems are configured, mapping has been built between interrelated entities
Configuring an intermediate layer often requires a lot of mapping work. For example, when the source system has 200 attributes and the receiver system has 100, some of them are composite, and some are taken from another source system. Or systems contain different types of data and fields. All the logic must be built and taken into account so that the data is not lost and enters the receiver systems in the required amount.
- The most complex stream has been developed
The most time-consuming part of the development phase is implementing the most complex flow. In fact, all other system flows are built on it, pitfalls are identified, and infrastructure restrictions related to setting permissions and publishing services on development stands are solved.
If the team missed or did not do enough work on the pre-project phase, this will begin to affect the speed and quality of ESB implementation right now.
Thus, in the project for a startup working with taxi companies, the KT.team also skipped the pre-project stage. As a result, the connectors had to be redesigned three times to ensure that the architecture worked. In the first version, all taxi fleet databases were surveyed using one service. This saved server resources, but did not ensure fault tolerance: any error would stop all subsequent requests.
In the second version, the bus was connected to each database through a separate service, which prevented a massive failure due to one inaccessible database, although it required more resources. But this solution involved many additional steps, for example, collecting and logging all errors.
The final version of the architecture was also based on separate services for each connection point, but they all worked according to the same scheme. This made it easier to add elements to the system: to add a new taxi fleet, you need to copy one of the existing services and change the mapping slightly. This task did not require employees to have a deep knowledge of the code. As a result, the implemented service bus allowed the startup to connect more than 10 taxi companies in 2 months.
The stage of creating documentation to support the solution
Average duration: 25—30 hours (depending on project complexity)
In order to support and scale the solution, the customer needs tools that will allow them to quickly make changes to the system. The ESB concept itself is designed to facilitate this work: while in monolithic systems, a change in one meant the need to manually refine all the others, the intermediate layer allows you to transfer this process from development to configuration. All you need to do is change the settings to support the integration with the new introductions.
In order to make it easier for the customer's employees to support the system, at the final stage of the project, the implementation team prepares detailed documentation for building integrations: where to go to perform the required action, what blocks, elements and subprograms to use, how to check the performance of integrations, etc. This phase is partly parallel to implementation and configuration.
Detailed documentation allows the customer's team to independently configure new integrations and support old ones, including through the efforts of citizen developers — experienced users who are not developers.
CVs
- On average, the implementation of an ESB layer takes about 1.5-2 months before the MVP is launched, excluding the pre-project stage. However, in practice, the duration varies greatly from project to project and depends on the complexity of the current and planned solution architecture, the emergence of new inputs or customer requirements during implementation, and the number of specialists in the implementation team.
- Usually, the most time-consuming stages are the pre-project stage (analytics and architectural design of the future solution) and the development of the most complex flow.
- The speed, efficiency of project implementation, and the correct selection of tools depend more on the depth of development of the pre-project stage.
- An established business process for integrations on the part of the implementation team will help to see the solution's bottlenecks before development begins and avoid rollbacks and delays.
