PharmAssist — Prescription OCR Pipeline
Turns handwritten medical prescriptions into structured, validated healthcare data through a chained segmentation → recognition → NLP-correction pipeline.
- Context
- AbaNexus AI · Applied research
- My role
- ML engineer — model pipeline & serving
- Primary stack
- FastAPI · PyTorch · Mask R-CNN
Measured impact
- Recognition accuracy
- 90%
- On handwritten prescription text
- Architecture
- Hybrid
- Mask R-CNN segmentation + MobileViT recognition
- Inference
- Real-time
- Low-latency FastAPI serving layer
The problem
Operational challenge
Handwritten prescriptions are the worst case for OCR: inconsistent handwriting, domain-specific vocabulary, and a cost of error that is measured in patient safety rather than characters.
The approach
Implementation strategy
I split the problem instead of asking one model to solve it. Mask R-CNN isolates regions, MobileViT decodes them, then a KenLM language model and SymSpell correction pass constrain the output to a medical lexicon — so implausible drug names are corrected rather than emitted.
Engineering trade-offs
MobileViT over a larger transformer backbone was a deliberate accuracy-for-latency trade. The NLP post-processing recovers much of the gap at a fraction of the inference cost, which is what makes real-time serving viable.
What I built
Delivery highlights
- Chained three models into one pipeline with a single FastAPI serving surface.
- Constrained decoding with a domain language model, materially reducing implausible outputs.
- Built multi-modal intake (upload, clipboard paste, webcam) plus batch gallery review.
- Exports structured JSON so downstream healthcare systems can consume results directly.
How it fits together
System architecture
Interface
System screens
7 captures from the live system. Select any image to view it full size.