What is ESB in Simple Words
We often come across some weird opinions about ESBs. They say that “It’s something huge”, or “We don’t need it because we have microservice architecture”. Nevertheless, ESB is a pain-remover for any integrations you have in your architecture. In this article we’ll use simple and straightforward example to prove it.
ESB is an enterprise service bus, also called iPaaS (integrational platforms as a service)
ESB is a set of solutions to provide integrations. It necessarily includes monitoring, queue broker, and a low code jobs builder. In fact, it is a whole layer configured to solve the problem of data exchange. It can be a cloud solution, or a self-hosted system deployed on your servers – the choice will depend on the size of your company and size of development teams.
How to Make an Integration without ESB
Let's say you need an integration between the CRM system and the online store. You agree with the CRM system on the API format and then develop the extensions.
What will your team do if some integration breaks down? For example, the CRM stops receiving information about customers. They'll try to find out who's responsible for that. They’ll need to develop and start recording logs in e-Shop and try to move the customer from e-Shop to CRM. And the logs will show that everything is sent to the CRM perfectly correct.
Then you’ll contact CRM team. CRM team will claim that there are no problems with the system. After a while you’ll nail down that the problem is with a particular customer attribute – it is not validated. For example, some Chinese customer has only two letters in his surname, and systems accept only three or more letter names – and this has never been checked.
You need to change the logic in e-Shop. After fixing the error, you need to resend the customers to the CRM. But your integration doesn't have such a functionality and you need to develop a simple script and resend data.
Problem-Solving in Traditional Integrations
When do you detect the problem? When it already affects your processes. By that time you have already lost some money or reputation. But you can’t appoint some random employee to monitor the flows because it’s usually quite technical job and developers are still needed for it.
How fast can you solve the problem? First you’ll need to make a logging process and code for it, then make a debugging, then figure out what’s wrong, then you need to resend the script... In general, it’s not fast!
How fast can your e-Shop solution be changed? Also not fast, because some of your developers from e-Shop team have to spend their time supporting integration flows that don’t give you a business value. Valuable resource are wasted not on hypotheses that will increase your sales but on support of some piece of code.
Will that affect your ability to change CRM? Indisputably, yes. When you’ll want to update your CRM you will think twice and have to push your team to test this flows together. We came across a lot of teams that have a habit of staying to work on Friday evenings and spending weekends to make an integration work after every other update or change!
I must say that doing the same type of routine task, e.g. looking for errors in the integration, adding logs to check messages and calls, is extremely exhausting. The developer quickly gets tired of this. They think that instead of a cool technology they are engaged in messaging – “Am I the mailman?” They burn out and change jobs.
And the code abandoned by them becomes a real challenge for those who come after them. Imagine that you got a car that was made by a craftsman in a garage in a single copy and half of the parts for it were made in the same garage. Exclusive, expensive and unnecessary. With the code it’s all the same.
Summary on Traditional Integrations
- You spend money on integrations that can be supported only by developers.
- You find problems only when they already have the real affect on your business.
- Your development team works slower.
- Every integration is done slowly and without a clear result.
- Your systems are too complicated.
One System and Tons of Logic
What happens if you add one more system to your architecture?
For example, you decide that you need to send enriched data on your orders and customers to ERP. Your ERP system requires customer’s name to have maximum eight letters. So you need to adap existing logic to it and build many irrational, difficult-to-support integrations. You have only 3 systems, only two object that has to be in your systems – orders and customers. And how many integrations will we have?
You have a development teams that work with their given systems. Each team has its own backlog and its own metrics.
The CRM team has tasks for improving customer process and marketing mechanics for them.
The e-Shop team works with customer orders. Their objective is to increase number of orders and order processing.
The ERP team has tasks, for example, with inventory.
When you need integrations, a lot of integration tasks appear in the teams' backlogs. To ensure data transfer teams often reinvent the wheel. And each team does it in its own system. Sometimes it leads to duplicating parse, monitor, logging logic in the systems. In some cases just one team takes all integration tasks.
As a result, each integration depends not on your business logic but on tons of internal matters. Which development team has a shorter backlog? Which team promises to build an integration faster? etc.
Instead of understandable architecture you get something like this. Hard to understand, hard to support, hard to solve any problem.
No matter what kind of software architecture is in each particular system, microservices or otherwise. No matter what systems you need to integrate. You always get the same picture.
Entangling the Tangled
Is there any alternative? Yes – ESB! How will you manage with all that task with ESB?
The flow order changes this way:
- Customer information is always taken from e-Shop. The e-Shop knows only the attributes it has for Customer. You don’t need an integration code because it has a generic API. Your e-Shop team has to configure only integration with ESB often with no code – just in GUI!
- CRM and ERP make their own integrations to take Customers information when needed. The CRM team is responsible for its own integration with ESB, but nothing more is needed. No additional logic appears in CRM – it all is in middleware. Same thing with ERP.
Now who will detect the problem? ERP. When it receives data from e-Shop and faces something unusual, it will alert a monitoring system that always goes hand by hand with most ESBs.
If your CRM will claim it doesn’t receive the customer, all you need is to find the corresponding record in ESB. If it’s there, then problem is in CRM or in connector, made by CRM team. They don’t have to communicate with any other team to find out where the problem occurs.
And if customer is missing in ERP queue, then CRM and technical support team will ask e-Shop team to solve the issue, as it obviously occurred in their area of responsibility.
By the way, in most cases such a problem can be solved by technical support crew without developers’ help. It became possible because all integrations are developed in GUI or in a simple code.
By adding ESB to your architecture, you move the tasks of ensuring data transfer to a separate team, freeing up the backlogs of teams developing the rest of the system. And if you don’t have a separate ESB team – for example, you’re small business – your development team receives a tool that enhances their efficiency in a long-term.
Summary on Using Integrational Platforms
- Integrations are supported by separate technical operators, NOT by developers.
- You can fix anything before the problem starts affecting your business.
- Your development team works builds the product business value. All your systems have less code in them.
- Every integration is done as fast as it is possible.
- Clear vision of the result. You have simple and straightforward areas of responsibility.
Few More Words about ESB
- Some techies think that ESB and brokers are one and the same. No, brokers lack tools for building an end-system connectors.
- There are code-based ESBs on market, but we don’t recommend them. The reason for implementing ESB is to enhance speed and supportability of integration, and it is achievable only with ESBs with GUI for creating a microservices.
- Microservice architecture and ESB don’t conflict! Microservice application has a bounded context. What is that exactly? CRM can calls your customers “Merchants”, e-Shop can call them “Customers”. Customers in this systems can have different attributes and so on. One of the functions of ESB here will be to translate between contexts. Of course you don’t need ESB inside microservice app – broker would be enough.
- You can't define service boundaries unless you use process management approaches. We'll talk about that in one of our next articles.