LLM Gateway: implementation and integration

45-85%lower LLM cost by routing up to 85% of requests to cheap models while keeping 95% of frontier-model quality

€1.15Btotal GDPR fines in Europe for 2025 — a direct risk when sending PII to third-party LLMs

40%+share of requests served by the semantic cache in production — savings with no loss of answer quality

53%of AI teams exceed their LLM cost forecast by 40%+ when scaling without a gateway

AI layer

An AI assistant must take action, not just reply with text

Core thesis of the AI block: a pilot with measurable impact, private data under control, agent actions logged, quality passes evals before scaling.

1-2 hof daily routine the assistant takes off an employee or manager

2-4 wksenough for a pilot on one process with an impact metric

40%of agentic AI projects Gartner expects to be canceled without clear value

Assistant ≠ chatbot

A chatbot answers; an assistant checks the regulations, queries systems, records the deviation and proposes the next step.

Control plane

Agent registry, owner, permissions, memory, evals, trace logs, kill-switch and budget at the enterprise-layer level.

Data

RAG returns an answer with a source citation; LLM Gateway obfuscates personal data before the model and restores it after the response.

Processcorporate memoryagentaction in the systemlogs and evals

Internal links

Agent governance layer RAG with sources LLM Gateway and personal data

Sources

Anthropic: Building effective agents Gartner agentic AI cancellations Chip Huyen: AI Engineering

Industry solutions

What you can do with LLM & Security Gateway

All Solutions

Banking and fintech Connect an LLM to support and scoring with obfuscation of names, accounts and cards before sending to the model Processing customer cases and requests without sending banking secrecy data to third-party modelsLearn more →Insurance Automate case triage and claims settlement with anonymization of policy and policyholder data Initial processing of applications and customer correspondence with no risk of PII leaking into the LLMLearn more →Healthcare and medtech Apply an LLM to medical records and patient requests with obfuscation of diagnoses and identifiers Preparing discharge summaries and answering patient questions without passing medical secrecy data to the model providerLearn more →Retail and e-commerce Route the flow of support and product-description requests between cheap and frontier models Customer support and content generation under a controlled LLM budgetLearn more →Telecom Set one gateway for all AI services with per-team limits and cost attribution Manage the cost and reliability of dozens of LLM services in one placeLearn more →Industry and B2B manufacturing Connect an LLM to technical documentation and service requests with fallback between models Supporting engineers and handling tickets with no service downtime during a provider outageLearn more →Public sector and education Deploy a self-hosted gateway with in-perimeter obfuscation of citizens' and students' PII AI assistants for cases and monitoring without sending personal data outside the perimeterLearn more →Logistics Apply an LLM to orders and tracking with anonymization of recipient addresses and contacts Processing orders and customer requests with cost control and PII protectionLearn more →

Capabilities

LLM & Security Gateway capabilities

App / agent: a request with PIIPII detector: names, phones, addresses, cardsObfuscation: PII → stable tokens + session mapGateway: routing, budgets, semantic cache, guardrailsLLM provider: receives only anonymized textModel response: with tokens instead of PIIDe-obfuscation: tokens → original values via the session mapUser: a response with real data; PII never left the perimeter

A two-way PII protection contour in LLM & Security Gateway. The application request passes through a PII detector that replaces personal data with stable tokens and stores a session mapping. The anonymized request, with routing, budgets and cache applied, goes to the chosen LLM. The model response returns with the same tokens, is de-obfuscated via the session map — tokens are substituted back to their original values — and the user gets a response with real data that never left the perimeter and never entered the provider's logs.

PII obfuscation before sending to the model

Names, phones, addresses and card numbers are replaced with stable tokens before sending; the model and its logs never get real customer data, removing GDPR fine risk

De-obfuscation in the response

Tokens in the model's response are restored to original values via a session map — the user sees real data that never left the perimeter

Model routing by price and quality

Cheap requests go to light models, complex ones to frontier; costs drop 45-85% with no noticeable quality loss

Budgets and limits per team and project

Token limits per key, team and project stop overspend before the provider bill — an end to surprises of 40% over forecast

Fallback and load balancing across providers

If one model fails or slows down, traffic shifts automatically to a backup — the AI service does not go down with the provider

Observability and cost attribution

Every request is logged with model, tokens, latency and cost via OpenTelemetry — you see who spends how much and where quality degrades

Semantic cache

Repeated requests are served from cache (40%+ hit rate in production) — lower bill and latency with no loss of answer quality

Unified key and access management

Provider keys are stored in the gateway; teams get virtual keys with instant revocation — secrets never leak into application code

Guardrails on input and output

Prompt injection detection and content filtering before and after the model reduce leak risk and toxic responses in production

Approach

How we implement LLM & Security Gateway

Minimal core modification

We don't fork or patch the LLM & Security Gateway core. LLM & Security Gateway stays on the standard, upgradable version — we move business logic into separate microservices alongside it, so platform updates don't break your customizations.

International Standards, Not Homegrown Hacks

Where a mature international solution exists, we use it instead of inventing our own protocol or platform. Before writing code, we study how the problem is already solved in the industry.