Client
Electroresheniya LLC, operating under the EKF brand, develops, manufactures, and sells electrical equipment and comprehensive energy-efficient solutions for industrial enterprises. EKF works with the largest CIS corporations in the oil and gas industry as well as manufacturing, logistics, and transportation companies, and its products are sold in 20 countries. The scale of the business is constantly growing, which inevitably affects its IT landscape: the volume of data moving between systems and the number of integrations are increasing.
Objective: ensure up-to-date data in all systems and fault-tolerant architecture
The EKF team wanted to prevent scaling issues in its IT architecture. As the volume of data grew, the following challenges could arise:
The EKF team concluded that Kafka message broker needed to be implemented and entrusted with data transfer between systems and quality control of message delivery.
EKF turned to KT.Team with this task.
By interviewing the EKF team during the pre-project phase, KT.Team clarified the client’s request: to establish a process for keeping data up to date in all systems and achieve fault tolerance in the IT architecture.
This kind of task can indeed be solved by implementing an ESB layer, not just a message broker. However, to ensure that the listed problems do not arise in the new architecture, the integration logic must be planned correctly.
Therefore, together with the client, we divided the project into two stages:
- slow delivery of data updates, the delay between updating information and its arrival in all consuming systems, began to have a negative impact on the company’s business processes;
- delays in detecting, analyzing, and resolving issues in data transfer between systems;
- data loss or delayed data updates.
- analysis and planning stage;
- implementation stage.
Result 1: defined system areas of responsibility
KT.Team’s previous experience on ESB implementation projects showed that companies often do not have an up-to-date and complete as-is map of flows and integrations. Each business unit leader and senior IT specialist has expert knowledge of how relationships are structured within their own area and with their own system. They have only a general understanding of the other parts of the integration map, which is enough to cooperate with other departments.
The absence of a complete as-is map is not critical for the organization. However, having one will give the IT department a single reliable source of knowledge about possible weak points in the current architecture, ways to prevent issues, and possible ways to improve the exchange architecture.
That is why we spent the first 10 meetings with the EKF team building and refining the as-is diagram. Within this diagram, we defined the areas of responsibility for all systems included in the EKF environment.
During the pre-project assessment, the EKF team and KT.Team identified which system was the source for each type of information and which was the recipient. This made it possible to pinpoint where information could be duplicated or conflict with the data in the source system. Slower information exchange could have caused additional issues.
In addition, KT.Team consultants identified processes in which information reached the receiving system through several third-party systems. These processes were more prone to failures and errors than the others, and investigating the related incidents required significant time.
We'll curate materials for your task
We'll reply within 30 minutes and send relevant cases, diagrams, or analyses tailored to your context.
Result 2: the client team aligned its understanding of the current state of the company’s IT environment
At the start of the project, each key EKF IT specialist had deep expertise in their own area, but there was no complete, reliable, and publicly accessible map of the company’s IT environment.
After 10 meetings, the KT.Team and EKF teams jointly built an integration map current as of the project start, combining and enriching the local diagrams previously prepared by EKF employees.
By forming a complete picture of the IT environment, EKF's key IT specialists gained a shared understanding of what the company's IT landscape looks like and what potential issues it has overall, rather than only in individual areas.
The resulting scheme helped capture what complicates the data exchange process:
The analysis and documentation of the as-is state confirmed the IT environment’s dependence on central systems. The complexity of the current integrations made it practically impossible to replace any of the central elements because of the high risk of data loss and loss of functionality across the entire IT environment. Such a replacement would have required accounting for hundreds of nuances in the existing integrations.
- there is a single point of failure for customer-facing services (the distributor portal, the online catalog, etc.);
- data is duplicated across several systems and proxied, meaning one system acts as an intermediary when transferring information between other systems, and there is no single owner for the information domain;
- the main ERP system is overloaded, including with data loading tasks, which causes errors and slows down data exports;
- different exchange mechanisms are used for different systems and use cases, which increases the complexity of integration support and incident handling given the scale of the IT environment;
- Because the systems are tightly interconnected, some business processes in one system cannot be performed if another system is unavailable. For example, a distributor cannot place an order when the ERP is down.
Result 3: designed a new integration map with loose coupling and high fault tolerance
Even in the early stages of discussions with KT.Team, the EKF team described a number of common cases from their day-to-day work that they had to handle regularly. For example, if the ERP was unavailable for any reason, distributors could not place an order on the portal because they could not see the assortment.
Such issues can be prevented through asynchronous data exchange and separate storage for different types of information.
The new integration architecture was designed in line with these principles. For each system, domains, meaning its areas of responsibility, and the types of data for which it is the source were defined.
Information is transferred asynchronously: first to the storage layer, then to the receiving system. This made it possible to eliminate the mutual dependency between systems in the IT environment.
When designing the new IT environment together with the EKF team, we discussed exactly how data storage would be organized, how microservices would be separated, and how their interaction would be structured.
At this same stage, the consulting team:
The consulting report set out the principles for monitoring the state of systems and data flows, the incident notification mechanism, and the list of information sent to IT specialists to fix errors. EKF specialists gained a shared view across all teams of how the IT environment would work, how teams would learn about errors, and how undelivered messages would be stored.
- The systems were relieved of the function of mutual data exchange, including intermediary transfer. As a result, update exports are accelerated many times over, and the delay between updating information in the source system and loading it into the consuming systems is reduced.
- The monitoring and logging systems built into the ESB make it possible to record incidents as they occur, including what happened to the data and when, and promptly notify technical support staff. This makes it possible to address detected issues in time.
- Data loads and exports happen based on triggers or at a fixed frequency, depending on the business process. Each message path is tracked automatically, so information will not be lost.
- clarified for the client's IT specialists the requirements for service independence (why they were formulated this way; how service independence would affect the company's business processes and the workload of IT specialists);
- prepared a step-by-step description of the transformation process for each connector and flow, including the reasons for all the changes described and the expected effect of implementing them;
- explained the logging scheme, monitoring setup, and technical support organization.
Result 4: provided a comparison of four ESB solutions, enabling the client to choose the system based on the parameters most important to them
The KT.Team team prepared a comparative analysis of tools that could be used to configure the integration the client needed. In the first stage, four solutions were compared: DATAREON, Mule, Talend, and WSO2, across 50 key parameters, including the adapter development language, availability of a library of ready-made connectors, fault tolerance, support for various formats and protocols, role and access management, scalability capabilities, and more.
In the second stage, each parameter of the two finalist applications was scored. Based on the overall ranking, Mule was selected for project implementation.
Result 5: split the ESB layer implementation process into stages with a clear duration and value for each
The KT.Team team developed an ESB layer implementation roadmap with a justified duration for each stage in hours, for each connector and flow separately. The implementation was split into several stages, and after each one EKF receives clear value and a functioning IT environment. When planning the implementation stages, the capabilities of EKF’s internal team were considered, and the budget for engaging outsourcing teams was estimated.
The stages were defined so that each one could be introduced safely for the company's IT environment and the previously implemented flows.
The effort for each stage is specified in hours, allowing EKF to compare this estimate with proposals from other integrators and choose the right contractor.
Together with the EKF team, we identified the key stage at which the bus implementation reaches ROI. To do this, we defined the block of integrations critical to the main business processes, changes to which would deliver immediate tangible value.
This allowed the EKF team to determine the required budget and the priorities for further work.
