A pilot is usually built on one provider's API and one large model, which is enough for a demo. In production, this setup reveals three problems.
A large model responds slowly when prompts are long and there are many users. During peak hours, the provider throttles requests, rejecting or delaying them because capacity is limited. And an expensive generative model spends tokens on tasks where it only needs to choose one of five options.
Each of the three AWS guides addresses one of these problems. In
Bedrock is unavailable in CIS, but the engineering challenges are the same, so for each guide I note what can be applied here. ###
One model across two nodes: prefill and decode separated. An LLM response is built in two stages.
In the first one, prefill, the model reads the entire prompt in one pass: the instruction, RAG documents, and conversation history.
This stage is constrained by GPU compute capacity.
In the second one, decode, the model generates the response one token at a time, making memory bandwidth the bottleneck.
If both stages run on one GPU, one user's long prompt slows generation for everyone else. In the AWS guide, the model is deployed in Kubernetes (Amazon EKS) across two GPU nodes using Ray LLM containers.
One node performs only prefill, while the second performs only decode.
Externally, it is a single OpenAI-compatible endpoint, so the application code does not need to change.
The same approach works for tensor, pipeline, and data parallelism when a model is split by layers or replicated across multiple machines.
The business takeaway is this: if you use RAG over internal policies and the prompt spans dozens of pages, response latency depends more on how these two stages are separated than on the model size. ###
Regional capacity runs out during peak hours
The second guide states the problem directly: when demand in a region exceeds local capacity, requests start being throttled. AWS addresses this through global cross-Region inference.
The request enters your region, Bedrock sends it where GPUs are available, while monitoring and access permissions remain in the originating region.
GPT-6 Sol and GPT-6 Luna are available from five regions (Thailand, Malaysia, Singapore, Indonesia, and Taiwan), while GPT-6 Astra is available from Singapore. A CIS company faces the same goal, but different constraints: under GDPR, personal data cannot be sent anywhere.
That is why we build the routing ourselves in our projects.
The request passes through the LLM & Security Gateway: the gateway masks personal data, selects a model based on the task type, and on failure switches the request from GigaChat to YandexGPT or to the customer's own Qwen or Llama deployment.
The application works with a single API and does not know which model responded to it. ###
Many tasks do not need a text generator
I consider the third guide the most underrated.
AWS identifies a class decision models - models that return scores for a fixed set of options instead of free-form text.
Such models route requests, choose the agent's next action, and apply policy decisions.
They are quantized, meaning their weights are compressed to 4–8 bits, and run through llama.cpp on ordinary CPUs. In our product catalog projects, such tasks are common: assign an item to a taxonomy node, fill a facet from a list of allowed values, and decide whether a tag needs a manual override.
All of these are closed-list classification tasks.
Sending it to a flagship model is like calling an architect to choose the color of a folder.
A small CPU model responds faster and costs less, and since it does not need a GPU, it is easier to deploy within the customer's perimeter.
