How is ESB Related to Software Speed and Quality

We often come across some weird opinions about ESBs. They say that “It’s something huge”, or “We don’t need it because we have microservice architecture”. Nevertheless, ESB is a pain-remover for any integrations you have in your architecture. In this article we’ll use simple and straightforward example to prove it.

ESB is a set of solutions to provide integrations. It necessarily includes monitoring, queue broker, and a low code jobs builder. In fact, it is a whole layer configured to solve the problem of data exchange. It can be a cloud solution, or a self-hosted system deployed on your servers – the choice will depend on the size of your company and size of development teams.

Let's say you need an integration between the CRM system and the online store. You agree with the CRM system on the API format and then develop the extensions.

What will your team do if some integration breaks down? For example, the CRM stops receiving information about customers. They'll try to find out who's responsible for that. They’ll need to develop and start recording logs in e-Shop and try to move the customer from e-Shop to CRM. And the logs will show that everything is sent to the CRM perfectly correct.

Then you’ll contact CRM team. CRM team will claim that there are no problems with the system. After a while you’ll nail down that the problem is with a particular customer attribute – it is not validated. For example, some Chinese customer has only two letters in his surname, and systems accept only three or more letter names – and this has never been checked.

You need to change the logic in e-Shop. After fixing the error, you need to resend the customers to the CRM. But your integration doesn't have such a functionality and you need to develop a simple script and resend data.

When do you detect the problem? When it already affects your processes. By that time you have already lost some money or reputation. But you can’t appoint some random employee to monitor the flows because it’s usually quite technical job and developers are still needed for it.

How fast can you solve the problem? First you’ll need to make a logging process and code for it, then make a debugging, then figure out what’s wrong, then you need to resend the script... In general, it’s not fast!

How fast can your e-Shop solution be changed? Also not fast, because some of your developers from e-Shop team have to spend their time supporting integration flows that don’t give you a business value. Valuable resource are wasted not on hypotheses that will increase your sales but on support of some piece of code.

Will that affect your ability to change CRM? Indisputably, yes. When you’ll want to update your CRM you will think twice and have to push your team to test this flows together. We came across a lot of teams that have a habit of staying to work on Friday evenings and spending weekends to make an integration work after every other update or change!

I must say that doing the same type of routine task, e.g. looking for errors in the integration, adding logs to check messages and calls, is extremely exhausting. The developer quickly gets tired of this. They think that instead of a cool technology they are engaged in messaging – “Am I the mailman?” They burn out and change jobs.

And the code abandoned by them becomes a real challenge for those who come after them. Imagine that you got a car that was made by a craftsman in a garage in a single copy and half of the parts for it were made in the same garage. Exclusive, expensive and unnecessary. With the code it’s all the same.

What happens if you add one more system to your architecture?

For example, you decide that you need to send enriched data on your orders and customers to ERP. Your ERP system requires customer’s name to have maximum eight letters. So you need to adap existing logic to it and build many irrational, difficult-to-support integrations. You have only 3 systems, only two object that has to be in your systems – orders and customers. And how many integrations will we have?

You have a development teams that work with their given systems. Each team has its own backlog and its own metrics.

The CRM team has tasks for improving customer process and marketing mechanics for them.

The e-Shop team works with customer orders. Their objective is to increase number of orders and order processing.

The ERP team has tasks, for example, with inventory.

When you need integrations, a lot of integration tasks appear in the teams' backlogs. To ensure data transfer teams often reinvent the wheel. And each team does it in its own system. Sometimes it leads to duplicating parse, monitor, logging logic in the systems. In some cases just one team takes all integration tasks.

As a result, each integration depends not on your business logic but on tons of internal matters. Which development team has a shorter backlog? Which team promises to build an integration faster? etc.

Instead of understandable architecture you get something like this. Hard to understand, hard to support, hard to solve any problem.

No matter what kind of software architecture is in each particular system, microservices or otherwise. No matter what systems you need to integrate. You always get the same picture.

Is there any alternative? Yes – ESB! How will you manage with all that task with ESB?

The flow order changes this way:

Now who will detect the problem? ERP. When it receives data from e-Shop and faces something unusual, it will alert a monitoring system that always goes hand by hand with most ESBs.

If your CRM will claim it doesn’t receive the customer, all you need is to find the corresponding record in ESB. If it’s there, then problem is in CRM or in connector, made by CRM team. They don’t have to communicate with any other team to find out where the problem occurs.

And if customer is missing in ERP queue, then CRM and technical support team will ask e-Shop team to solve the issue, as it obviously occurred in their area of responsibility.

By the way, in most cases such a problem can be solved by technical support crew without developers’ help. It became possible because all integrations are developed in GUI or in a simple code.

By adding ESB to your architecture, you move the tasks of ensuring data transfer to a separate team, freeing up the backlogs of teams developing the rest of the system. And if you don’t have a separate ESB team – for example, you’re small business – your development team receives a tool that enhances their efficiency in a long-term.