Word-Based Addition: A Test Before AI Deployment

Trigger

Simon Willison replicated Colin Fraser’s experiment on local hardware (DGX Spark).
The model adds numbers and returns the sum in words, while the numbers keep getting larger. Two years ago
Fraser ran the same test on GPT-4o, and the model often made mistakes. The new run
Willison ran it on Qwen3.8 27B (Willison’s analysis).
I am interested in the experiment setup.
A manager can run this test on their own task in one evening, and it will cost less than a single production error.

What we checked

The task looks childish: add two numbers and write the result out in words.
Frazer increased the difficulty step by step by adding more digits to the addends and plotted the share of correct answers.
Willison notes that GPT-4o could not use a calculator.
Its errors therefore show where the model’s ability to calculate independently ends. The test has an unambiguous reference answer.
Ordinary code calculates the total, and the strings are then compared.
There is no need to debate the “quality” of the answer. I rely only on the description of the experiment.
The results for Qwen3.8 27B are not shown in the excerpt, so I will not provide figures for them. See the original for the chart.

Where this occurs in business

A company implementing AI typically has three areas where the model works with numbers: - reconciling invoices and completion certificates in EDI: totals, VAT, and specification totals; - exports from 1C and Bitrix that the model summarizes in a management report; - product cards in PIM: dimensions, stock, and prices with unit conversion. A demo using three short examples in each area runs smoothly.

Failures typically appear with long numbers, large tables, and uncommon formats. The “accuracy versus input size” curve

Fraser’s experiment shows this boundary. The project budget is spent beyond it.

How to repeat the test

The test consists of four steps.

We choose a task whose reference answer can be generated with ordinary code.

We generate inputs of increasing complexity: longer numbers, more lines, and messier formatting.

Run the model in a controlled environment

Local deployment provides fixed weights, known parameters, and no hidden tools. Results are reproducible, and model versions can be compared with one another.

We plot the results and find the threshold beyond which the error rate is no longer acceptable for the business. The initial test requires a script of a few dozen lines and an engineer’s evening. If the threshold is below real-world volumes, the task is delegated to deterministic code. The model remains responsible for understanding the request and formatting the result.

Discuss your challenge with an architect

clients@kt.team Telegram @kt_team_it

The boundary of responsibility

The model understands intent well, parses unstructured text, and selects the appropriate call. Arithmetic, reconciliation, and aggregation are more reliable when delegated to code. Architecturally, it works like this: the model receives tools through MCP, calls a calculation function, and returns the result to the user. The calculation itself is performed by code. The second layer of protection is validating the response before it reaches the accounting system. Our LLM & Security Gateway includes an output validator that checks totals and formats and rejects suspicious results.

For document-related questions, we use RAG, so figures come from the source rather than being reconstructed from the model’s memory.

Model selection goes through the same test

Around the same time, Moonshot AI released Kimi-K3: 2.8 trillion parameters, open weights, and prices of $3 and $15 per million input and output tokens.
The license is non-standard and requires a commercial agreement if MaaS revenue exceeds $20 million.
For model selection, these factors matter no less than model size.
The model catalog is growing rapidly, while press releases are poorly suited for comparison.
It is more reliable to measure the share of correct answers on your data and the cost per correct result.
The same applies to the environment where the AI agent runs. Felix
Felix Rieseberg of Anthropic writes that the previous version of Cowork ran the VM on the user’s computer while inference took place in the cloud.
Users disliked the disk and battery usage, and work stopped when they closed their laptops. In the new version, both inference and the VM run in the cloud, with a separate sandbox for each session.
The execution environment affects results and total cost of ownership just as the model does, and it should be measured in the same way.

Conclusion

AI that does not move a metric is just theater, and management pays for the theater. The Fraser and Willison test sets the order: first establish a reference and a complexity curve, then decide whether to implement. An interface where users enter a request and receive the correct total depends on test sets, calculation tools, and output validation. A model that confidently writes the wrong total in words costs more than one that delegates the calculation to code.

Source

Simon Willison, “Qwen3.8 27B addition in words,” simonwillison.net. Original experiment with GPT-4o: Colin Fraser (Bluesky). Willison’s notes on Kimi-K3 and Felix Rieseberg’s quote about Cowork were also used.

Word-Based Addition: A Test Before AI Deployment

Trigger

What we checked

Where this occurs in business

How to repeat the test

Run the model in a controlled environment

Discuss your challenge with an architect

The boundary of responsibility

Model selection goes through the same test

Conclusion

Source

Discuss the article: Word-Based Addition: An AI Readiness Test

Continue on the topic

Related Cases

Custom Akeneo modules for delivery

OSNO-VA: AI accountant

AI ingredient recognition by barcode

Related solutions

Articles on the topic

Related videos

News on the topic