Apache Kafka: implementation and integration

latency ~2 msMessage delivery at the network throughput limit

trillion messages/dayThe cluster scales to thousands of brokers and petabytes of data

>80% Fortune 100Share of the world's largest companies using Kafka

7 trillion/dayMessage volume in LinkedIn's production Kafka cluster

Integrations

Change one system without rewriting the rest

The integrations block must explain four ailments of rigid exchange: data loss, a cascade of rework, source overload and inconsistency. ESB, Kafka and n8n serve different tasks, not one hammer.

50flows in 6 months — a speed benchmark for an ESB project

48flows in production with a target loosely coupled architecture

16xfaster launch of standard integrations compared to code

ESB

Routing, transformation, guaranteed delivery and low-code support of legacy exchanges.

Kafka

Durable log: an event is stored, re-read, multiple consumers read at their own pace.

n8n

Fast orchestration of a process and AI steps where heavy event streaming isn't needed.

sourcedata contractESB/Kafka/n8nmonitoringconsumers

Internal links

ESB approach Kafka n8n

Sources

Enterprise Integration Patterns Martin Fowler: Event-Driven Confluent: Kafka vs ESB

Industry solutions

What you can do with Apache Kafka

All Solutions

E-commerce and retail A single order event stream that triggers warehouse reservation, payment, and delivery Order processing: eliminates stock mismatches and manual reconciliation between the storefront, warehouse, and ERPLearn more →Finance and banking Streaming processing of transactions and account events with replay for auditing Real-time operations monitoring and antifraud: millisecond responses instead of batch reconciliationLearn more →Manufacturing and IoT Collecting sensor telemetry into an event log with threshold triggers Equipment monitoring: predictive maintenance instead of reacting after downtime occursLearn more →Logistics and transportation Shipment tracking and status events available to all systems through the bus Supply chain tracking: a single order status without polling adjacent systemsLearn more →Telecom Streaming network and billing events for real-time pricing and alerts Network monitoring and billing: real-time stream processing instead of nightly batchesLearn more →Media and content A user event bus (clicks, views) for personalization Recommendations and behavior analytics: a near-real-time profile instead of delayed exportsLearn more →B2B distribution Event-driven integration of PIM, ERP, and partner storefronts through a single broker Catalog and pricing sync: a change is published once and distributed to all consumersLearn more →Healthcare Event exchange between HIS, laboratories, and the patient portal Patient data coordination: results are available to adjacent systems without direct integrationsLearn more →

Capabilities

Apache Kafka Capabilities

Before: ERP↔CRM↔Warehouse↔Storefront↔Payments↔Analytics - an N×N tangle of direct connectionsEvery change requires updating several integrations at onceA failure in one system cascades through synchronous chainsAfter: all systems publish/read events through the Kafka busThe producer does not know the consumers, and the consumer can withstand a neighbor outageA new consumer (ML, reporting) connects to the stream without changing the sources

Comparison diagram with two panels. Left ('Before: point-to-point') shows 5-6 systems (ERP, CRM, warehouse, storefront, payments, analytics) connected by many crossing straight arrows forming a tangled web (N×N links, each brittle and requiring changes when modified). Right ('After: an event bus on Kafka') shows the same systems, each linked by one arrow to a central horizontal Kafka bus: producers publish events from above, consumers subscribe from below. Labels emphasize: left - coupling, manual synchronization, cascading failures; right - loose coupling, async, fault tolerance, decoupling, and adding new consumers without changing sources.

An event bus instead of point-to-point

One event stream instead of N×N direct integrations: adding a new system does not require touching the others

Loose coupling

Services are changed, replaced, and scaled independently - a release in one system does not break adjacent ones

Asynchronous processing

Peak loads are smoothed by the event buffer: the storefront does not go down when the warehouse or payment system responds slowly

Fault tolerance and replay

Events are stored in a durable log: a failed consumer catches up after recovery without data loss

Horizontal scaling

Rising load is handled by adding brokers and partitions without reworking the architecture

Real-time streams

Data becomes available to adjacent systems in milliseconds - orders, stock, and prices sync almost instantly

A single event log as the source of truth

New consumers (analytics, ML, reporting) connect to the existing stream without loading the source systems

Integration portability

Event contracts and a standard broker let you hand support over to another team or contractor without rewriting anything

Approach

How we implement Apache Kafka

Minimal core modification

We do not fork or patch the Apache Kafka core. Apache Kafka stays on the standard upgradable version, while business logic is moved into separate microservices nearby, so platform updates do not break your customizations.

International Standards, Not Homegrown Hacks

Where a mature international solution exists, we use it instead of inventing our own protocol or platform. Before writing code, we study how the problem is already solved in the industry.

Transferability

The solution is loosely coupled and documented: it can be handed over between teams and contractors without rewriting. You are not tied to us.

AI compatibility

Apache Kafka in the AI stack

A data stream for real-time ML and analytics

A single event log is a ready-made source for feature engineering, streaming scoring, and near-real-time marts without loading production systems.

A bus for AI agents

Kafka's pub/sub model gives agents a loosely coupled channel for exchanging events: one agent publishes a result, others react without knowing about each other.

Event sourcing for reproducibility

A durable log and replay let you replay event history for model retraining and AI decision auditing.

Event-triggered pipelines

A new event (order, request, stock change) automatically triggers inference or an agent workflow without polling systems.

News

What's new in Apache Kafka

All news

17.02.2026
Kafka 4.2.0: queues (share groups, KIP-932) are now production-ready
Share groups declared production-ready: consumers read the same partitions with per-record acknowledgment — a classic queue on top of Kafka.

Point-to-point with guaranteed delivery and confirmation is Datareon's standard bus model. Datareon →Queues and point-to-point messaging (JMS/Anypoint MQ) are a long-standing core MuleSoft pattern. MuleSoft →Queues with per-message acknowledgment and dead-letter handling were an ESB standard in Talend ESB long before KIP-932. Talend ESB →
17.02.2026
Kafka 4.2.0: the server-side Streams rebalancing protocol (KIP-1071) is now GA
Kafka Streams group management has been moved to the broker side: task assignment, topology storage, and observability are now generally available.
17.02.2026
Kafka 4.2.0: Dead Letter Queue for Kafka Streams (KIP-1034)
Kafka Streams exception handlers gained built-in routing of failed records to a dedicated queue (DLQ) instead of manual wiring.

Error handling via a separate error workflow is an n8n mechanism; DLQs have long existed in brokers and iPaaS. n8n →Dead-letter routing for failed messages was a standard ESB pattern long before Kafka Streams. Talend ESB →
04.09.2025
Kafka 4.1.0: queues (KIP-932) in preview, Streams rebalancing (KIP-1071) in early access
Release 4.1.0 brought share groups to preview (not for production) and added early access to the Kafka Streams rebalancing protocol (KIP-1071).