Context
A project with one of the top-3 developers in CIS explored automating support ticket classification with an LLM. The client had requests that staff sorted manually by team and service, so labeling quality and routing speed depended on human triage.
For the demo environment, KT.Team used a set of 12 thousand closed tickets from the first quarter. Before running LLM classification, the data was cleaned of junk, empty descriptions, long email threads and unusable labels.
Challenge
The goal was to show how manual ticket triage can become a managed AI process: with clear terms, controllable prompts, quality checks and the ability to refine rules based on classification errors.
A separate goal of the workshop was to agree on what data is needed for launch, how the learning logic works, where model instructions are stored and how the team will know that classification quality isn't degrading after changes.
Review a similar project with an architect
Solution
The approach is built as two-stage LLM classification. In the first step, the model uses the ticket's subject and description to identify the team that should handle the request. In the second step, the service is determined within the chosen team.
To improve quality, we built in a self-learning mechanism through rules and instructions. If a ticket is misclassified, the system analyzes the actual distribution, suggests adjustments to descriptions or triggers and runs control tickets to prevent regression.
- Prepared and cleaned historical tickets for the demo.
- Split classification into a team level and a service level.
- Defined the loop for adjusting rules after model errors.
Workshop
At the workshop, the team demonstrated the system on prepared tickets and discussed terms, the model, prompts, launch, Telegram notifications and the further rollout methodology.
After the meeting, the project had a working demo environment and an agreed path from manual ticket triage to AI classification, where rule changes can be tested on control requests.
Result
One of the top-3 developers in CIS received a demo of the LLM classifier on a real set of requests and a clear rollout methodology. It doesn't replace a production launch project, but it gives the team a proven basis for discussing data, prompts, classification quality and error control.
The next step is to refine terms, the learning approach and the rules for working with prompts in order to turn the demo environment into a stable support process.
