Replication Matters More Than Sharding in the AI Era

Kleppmann and Appleton explain why AI speeds up coding but does not replace data discipline, from replication to formal verification and design engineering.

  • The problem: code is faster than data
  • Kleppmann's thesis: the cloud made replication more important than sharding
  • How it works technically
  • Design engineering: where design meets code

The problem: code is faster than data

  1. Martin Kleppmann and Maggie Appleton, in two separate Pragmatic Engineer conversations, arrived at the same conclusion from different angles: AI has accelerated coding, but it has not eliminated engineering discipline—in data or in design.

  2. For business, this means speed is no longer the main constraint.

  3. The main scarcity is architecture that does not fall apart under real-world load. LLMs have reduced the time needed to write a function from hours to minutes.

  4. The data layer is not ready for this pace: schema migration, cross-region replication, and consistency under concurrent writes are tasks where errors remain invisible in a demo but are costly in production. A team that ships features at AI code-generation speed without revisiting its data model for the new load accumulates technical debt that will surface at the first traffic peak.

Kleppmann's thesis: the cloud made replication more important than sharding

  1. Kleppmann, author of "Designing Data-Intensive Applications," describes the shift this way: manual sharding is a technique from the previous decade, when infrastructure was expensive and inflexible.

  2. The cloud has made horizontal scaling more affordable through managed replication: Postgres replicas, Kafka partitions with automatic rebalancing, and managed Elasticsearch clusters.

  3. The engineer designs the consistency model: what happens when the network connection between replicas is interrupted and what guarantees a write provides. Second thesis

  4. Kleppmann goes further: AI could democratize formal verification by generating code and a proof of its correctness at the same time.

  5. Today, formal methods (TLA+, property-based testing) remain the preserve of a small group of academically trained engineers.

  6. If a model can write an invariant specification alongside a schema migration, the barrier to entry falls, and checking data for consistency gaps is no longer a luxury reserved for large companies.

How it works technically

The practical workflow looks like this: an LLM agent proposes a database migration → the same architecture generates property-based tests for edge cases (concurrent writes, transaction rollbacks, duplicate Kafka events) → a separate step formally verifies the invariant before deployment, rather than after an incident.

At KT.Team, we build this kind of architecture with Python and Apache Kafka for the event layer, Elasticsearch for search and aggregations, and MCP as the protocol through which an AI agent securely accesses data—via an LLM & Security Gateway that logs and limits every call, rather than connecting directly to the production database. This is AI-native integration: an architecture with checks at every step between the agent and the data.

Assess where AI can deliver impact in your process

Design engineering: where design meets code

Appleton describes design engineering as a discipline at the intersection of design and development. A design engineer takes the interface through to a production build, bypassing the handoff of a design mockup to a separate developer. AI accelerates this intersection: generating components from design tokens and UI variations for A/B testing takes minutes instead of days. The discipline remains manual: a design-token system that does not sprawl across a project requires a human architectural decision, not a prompt.

What This Means for Business

The TTU principle—time to use—applies here. The value of an AI tool lies not in how impressive its demo looks, but in how long it takes to go from a request to a result you can trust under load. A seemingly simple outcome—stable replication, a correct zero-downtime migration, an interface without visual bugs—requires complex engineering under the hood: data models and verification processes that will not collapse under real traffic.

Conclusion

AI removed the constraint on coding speed. The constraint on the correctness of data systems remains. Companies that use this opportunity to hire prompt engineers without revisiting their replication architecture and invariant testing will get fast code and slow degradation under load. Companies that embed formal verification and event-driven architecture into the same AI architecture will get a system that holds up on the first high-traffic Friday—the kind of outcome business actually calls a result.

Discuss the article: Replication Over Sharding: Data in the…

Enter your email or phone number so we can get back to you.

Send via: