Education
Classifying admission documents with a language model
A training centre received a few thousand applications each intake and sorted them by hand. I put a model into the workflow with a quality harness and a cost ceiling.
91% of documents classified correctly without a human
- Client
- Client asked not to be named
- Duration
- 2 months
- Engagement
- Fixed price
Context
Each intake, two staff spent about three weeks reading and sorting a few thousand applications into six categories. The work was repetitive, and the error rate climbed towards the end of each cycle through fatigue.
Management had tried an off-the-shelf automation tool and dropped it after two weeks: it could not explain why it had put an application in a category, so nobody trusted the output.
What was built
I started with the evaluation harness, not the model. Four hundred hand-sorted applications became the reference set before any model was tried against it.
The model returns a category together with its reasoning and a confidence score. Anything below the confidence threshold goes to a person instead of being guessed. Model spend has a hard monthly ceiling; past it the system routes everything to the manual queue rather than quietly spending money.
Data capture runs on a phone, at the place the work happens. The real user of this screen is standing up, busy and has one hand free, so it carries only the fields that are actually required.
Outcome
91% of applications were classified correctly with no human involvement, measured on the 400-item reference set rather than on data the model had already seen.
The remaining 9% land in the manual queue, which is the intended behaviour: the system says "I am not sure" instead of guessing. Three weeks of seasonal work became about four days.