Microservices or modular systems? How to choose an architecture approach for an IT product

Suppose your project involves complex warehouse logic.

If you choose a modular architecture, developers do not just need to build functionality for managing these warehouses - they need to override or extend the multi-warehouse module, which in turn relies on core methods.

At the same time, you need to account for complex warehouse return logic: dependence on events from the CRM system, moving goods between catalogs, and so on. You also need to account for hidden logic related to refunds, bonus points, and so on.

When so many overrides happen, the monolith changes substantially.

It is important to remember that the relationship between the amount of new functionality and the number of modules is nonlinear: adding one feature requires changes to several modules, each of which affects the others.

Or override a large number of methods from other modules in the new module, which does not change the essence of the problem.

After all the changes, the system becomes so complex that adding further customizations will require an unreasonably large number of hours.

Documentation for modular systems is hard to keep up to date.

There is a lot of it, and it becomes outdated with every change.

Updating one module requires changes to several other documents (user and technical documentation), and they all have to be rewritten.

As a rule, there is nobody to handle such work: spending valuable IT specialists' time on it simply means wasting the budget.

Even storing documentation in code (PHPDoc) does not guarantee its accuracy. In the end, if documentation can differ from the implementation, it inevitably will.

Another problem caused by code coupling is the difficulty of updating the platform. For example, Magento contains two million lines of code.

Wherever we look, there is a lot of code everywhere (Akeneo, Pimcore, Bitrix).

When adding functionality to the core, it is best to account for changes in your custom modules.

Let us take Magento as an example, as similar cases exist in any modular system on large projects. At the end of 2018, a new version of the Magento 2.3 platform was released. Multi-source inventory and Elasticsearch were added to the Open Source Edition.

In addition, thousands of bugs were fixed in the core, and some nice improvements were added to OMS. What did e-commerce projects face when they had already built multi-warehouse setups on Magento 2.2?

They had to rewrite a lot of logic in order processing, checkout, and the product page to move to the out-of-the-box functionality.

After all, this is the "right" way - why duplicate out-of-the-box functionality in modules?

Reducing the amount of custom code in a large project is always useful, since all out-of-the-box methods account for exactly these multi-warehouses; updating the product without such refactoring may be pointless (leaving security issues aside for simplicity, especially since they can be applied without upgrading).

Now imagine how long such an update would take.

How can this be tested without integration tests, which are hard to write? It is no surprise that for many, platform updates happen either without refactoring, but with more duplication, or, if the team wants to do everything properly, with a long detour into refactoring and cleanup.

One of the most important problems in project management is that the client does not see the full logic and all business processes of the project.

They can only be reconstructed from the code or from documentation, whose accuracy, as mentioned earlier, is difficult to maintain in modular systems. Yes, Bitrix has a BPM component, and Pimcore has workflow visualization.

But this attempt to manage modules through business processes always conflicts with the existence of a core.

In addition, events, complex timers, and transactional operations cannot be fully supported in a modular architecture system.

To repeat once again, this applies to medium and large companies.

For a small company, the capabilities of modular systems will be enough.

But if we are talking about the enterprise segment, this solution is noticeably missing a single control center where you can open any process or status on a diagram and see exactly how something happens, what exceptions there are, and which timers, events, and cron jobs exist.

There is no way to change business processes instead of modules.

Process-based project management is overwhelmed by the pace of change and logic coupling.

BPMS makes it possible to create a business process in which the project team (developer or business user) defines the sequence for launching microservices, as well as the conditions and branches that guide its flow.

At the same time, one business process, meaning a sequence of actions, can be included in another business process.

All of this is clearly represented in the BPMS: in real time, you can view these diagrams, edit them, and deploy them to production.

The principle of a self-documenting environment is applied here to the fullest extent - the process works exactly as visualized!

All microservices become process building blocks that can be added from the toolkit by a business user.

The business controls the process, and the developer is responsible for the availability and correct operation of a specific microservice.

At the same time, all parties understand the overall logic and purpose of the process.

The principle sounds very simple, and an inexperienced developer or business user may think that a BPMS does not improve the microservice design strategy itself. In other words, you can write decent microservices without a BPMS. Yes, you can, but it is difficult.

When a developer writes a microservice without BPMS, there is inevitably a temptation to cut back on abstraction.

Microservices become outright large, and sometimes even start reusing others.

There is a tendency to cut corners on making the transfer of results from one microservice to another transparent. A BPMS encourages more abstract code.

Development is carried out in a process-driven way, with inputs and outputs defined.

Imagine an order processing workflow: we need to go to a marketplace, collect all the good orders, and start processing them. Order queue processing diagram Take a look at the diagram (a portion of it).

Here it is defined that every 10 minutes we check all marketplace orders, then run the processing of each order in parallel, which is indicated by the vertical "hamburger" in Process Order.

If successful, we send all data to the ERP and finish processing.

If we ever need to pull logs for processing a specific order, in Camunda, JBoss, or any other BPMS we can fully restore all the data and see which queue it was in and what input and output parameters were used.

Imagine that an error occurred during product delivery.

For example (by the way, this is a real case), the shipping company picked up the order, and after that the warehouse burned down.

Another real story: the pre-New Year rush, and the goods were first delayed and then perhaps lost. In such a case, events are clicked together in the BPMS with the mouse, for example a customer notification if the delivery time has passed.

If we receive an error from the shipping carrier, we can trigger the process down our own branch and stop everything: notify the customer, offer a discount on the next order, or issue a refund.

Such exceptions are hard not only to program outside the BPMS (for example, a timer within a timer), but also to understand in the context of the entire process.

Imagine the same delivery order.

In total, we have three possible outcomes: the item was delivered on time; the item was not delivered on time; the item was lost.

In BPMS, we can define the procedure for handing goods over to a carrier and then wait for one of two events on the order: successful delivery (messages from the delivery process confirming that everything has been delivered) or a time-based trigger.

If the time has not passed, you need to launch another service: a review with an operator of this specific order (you need to create a task for them in the OMS/CRM system to investigate where the order is), followed by notifying the customer.

But if the order was delivered during the investigation, the investigation must be stopped and order processing completed. In BPMS, all interruptions and exceptions are handled by BPMS itself.

You do not overload the code with this logic, because having such logic in the code would make microservices large and poorly reusable in other processes.

Let's look at service orchestration using Camunda.

The following images will help you see how convenient it is to manage microservices with a BPMS and an orchestrator.

All processes are visual, and the logic is obvious. Business processes look like this:

Business process diagram in Camunda (BPMS with orchestration) Example (order, availability service):

Business process demo in Camunda: order, availability service. It is visible that this order had the "No stock" branch.

Another order instance (sent to picking):

Business process demo in Camunda: order sent to picking

The order moved on and, according to the decision table (DMN), went into the handling branch for a specific delivery operator (Boxberry):

Business process demo in Camunda: order handling by a delivery operator Transition to the nested process:

Business process demo in Camunda: nested process The nested process completed:

Business process demo in Camunda: nested process

Business process execution history:

Business process demo in Camunda: history

Properties of this kind of visualization: business processes are easy to read even for an untrained user; they are executable, meaning they work exactly as drawn; there is no mismatch between the "documentation" and the actual code behavior; processes are transparent: it is easy to see how a particular import, order, or processing step went through, and easy to understand where an error occurred. See also: