An AI assistant in a board meeting: where it breaks and how to fix it

In one week, AWS AWS's machine learning blog five technical analyses were published.

Four of them focus on the infrastructure around the model: cluster recovery after a failure, network throughput, storing data in another region, and answer validation.

The NarrateAI case is the most useful one for business.

It is an AI assistant that answers executives’ data-related questions during a business review.

Its authors state plainly: if an incorrect figure on the screen harms someone's reputation, a powerful model alone is not enough. This leads to a rule for executives.

An AI project's budget is split between the model and the validation layer.

In our experience, this infrastructure layer consumes most of the money and time.

The authors of NarrateAI identify four failures that appear only in production. Fabricated metric. The model writes "margin increased by 3 points"

, although no such figure exists in the data. The statement sounds plausible, and no one double-checks it during the meeting.

The error is discovered a month later, after the figure has already informed a budget decision. API limits (throttling).

The model provider limits the number of requests per minute.

Ten participants ask questions at once, and the assistant falls silent at the worst possible moment. Validation delay.

Each control layer adds seconds.

During a meeting, everyone notices a pause in the response, and people stop asking the assistant questions. Evaluative language. "Significant growth," "a weak quarter."

The model gives an assessment it was not asked for, while the executive is held accountable for the wording.

We build such systems with four layers, one for each failure. ### Code calculates the numbers

The model converts the question into a query to the data store: SQL to a data mart, or an API call to 1C or ERP.

The system does the calculations; the model only turns the result into text.

Data access runs through MCP, a standard protocol that lets the model call tools with predefined permissions. ### The validator checks the answer against

Separately, a dedicated step extracts numbers from the answer and compares them with the query result.

When the results diverge, the user does not see the answer: the system either generates it again or says, “There is no such data.”

The check runs in parallel with generation and adds almost no latency. ### The gateway handles the load

Requests to models pass through the LLM & Security Gateway.

It queues requests, caches frequent questions, switches to a backup model when the primary reaches its limit, and keeps personal and commercial data from leaving the system. ###

The dictionary prohibits evaluations without a threshold

The word “growth” is allowed only with a number and a time period.

The validator removes evaluative adjectives if the reference directory contains no threshold defined by the business.

An average CIS business does not train MoE models on hundreds of accelerators.

It connects Qwen, GigaChat, or YandexGPT through an API, or deploys an open model on its own server.

It has the same four failures listed for NarrateAI: the model makes things up, the provider imposes limits, validation slows things down, and wording becomes inconsistent.

The key metric here is TTU, time to use: the time from launch to the first decision made with the tool.

If an executive does not trust the assistant, TTU is infinite regardless of the model.

A one-line answer looks simple on screen.

Behind it are data queries, reconciliation, queuing, and a glossary, and someone will have to do this engineering work.

We apply the same principle in PIM projects when generating product card text, facets, and taxonomy.

The model suggests text or a category, rules check the attributes against the reference directory before publication, and people see only what has passed validation.