Document Classification for Claims Handling

The back office received a large daily volume of scanned and emailed documents — forms, receipts, correspondence, medical reports — all landing in one tray for staff to open, identify and route. Identification was most of the work and none of the value, and it created a backlog that grew whenever volume spiked. Misrouted documents were worse than slow ones, because a document sitting in the wrong queue is effectively lost until someone notices. They wanted automatic routing with a clear and honest fallback for anything uncertain.

I was the machine learning engineer on a team of four, with a backend engineer who built the service around my models, a business analyst, and an operations supervisor who owned the labelling. I owned the model, the evaluation and the confidence thresholds. I insisted on owning the threshold decision specifically, because that is where the business risk actually lives and it was being treated as a technical detail.

I built the document classification model, the OCR preprocessing chain for scanned documents, and the field extraction for the three form types that carry structured data. I owned the evaluation harness, including the per-class error analysis that showed us where the model was quietly bad rather than just how good it was overall. I built the confidence thresholding and the drift monitoring that alerts when the incoming distribution shifts. I did not build the service wrapper, the queue integration or the review interface.

The decision I argued hardest for was setting thresholds per document class rather than one global threshold. Misrouting a medical report has a very different cost from misrouting a receipt, and a single threshold pretends otherwise. It made the configuration more complex and let us be aggressive where errors were cheap and conservative where they weren't. I also chose to fine-tune a modest pretrained transformer rather than call a hosted model, because the documents can't leave the client's environment and running our own also made the latency predictable. For scanned documents I put real effort into deskewing and denoising before OCR, which felt like a detour and improved accuracy more than any modelling change did.

No document could leave the client's infrastructure, so hosted services were out and everything ran on their own hardware, which was modest. Labelled data was scarce at the start — we had about eleven hundred labelled documents, which the operations team grew during the project. Scan quality varies enormously because several sources are fax-era workflows, and I could not improve the source. There was also a hard requirement that the system never silently discard a document, which shaped the fallback design.

About 78 percent of documents now route automatically, with the rest going to a review tray that staff work through far faster than the old undifferentiated pile because the system shows its best guesses. Average time from arrival to reaching the right queue fell from most of a day to under fifteen minutes for the automated share. Misroutes are lower than the manual baseline the operations supervisor measured before we started, which was around four percent and is now near one. The drift monitor has fired twice, both times a source changing its scanning setup, and both were caught before accuracy degraded noticeably.

The operations supervisor organised the labelling and had a much better instinct than I did for which classes actually confuse people, which is where I focused error analysis. The backend engineer built the service and the review tray, and pushed back usefully on an API design that would have made his retry logic impossible. The business analyst worked out the relative cost of each misroute type, which is what made per-class thresholds defensible rather than arbitrary.