Integrating a Custom Trained Model into Your Receipt Scanning App

In the previous blog, I shared how to train a custom model using CreateML and your own dataset. If you haven’t read it yet, be sure to check it out first. My goal with this blog is to help you build something truly useful without feeling overwhelmed by terms like training models or using Vision. This guide is designed to be simple to follow and easy to expand with additional features as you wish.

In this blog, I’ll walk you step-by-step through how to create a smart receipt scanner with these key features:

Key Features

• Scan or Upload Receipt Image (Camera or Gallery)

• Extract Text from Image (OCR with Vision)

• Parse and Structure Extracted Data

• Categorize Items (Using self trained ML model)

• Display Results by Category

To start, I’ve created a ReceiptScannerView using SwiftUI, which allows users to upload a receipt image from their photo gallery.

I’ve implemented Scan or Upload Receipt Image feature using the PhotosPicker.

To Extract Text from Image I utilised Apple's Vision framework. The following function, recognizeText(from:completion:), performs OCR (Optical Character Recognition) and returns an array of detected text lines:

What this function does:

• Convert UIImage to CGImage:
  Vision requires a CGImage to work. If conversion fails, the function simply returns an empty result.

• Create a Text Recognition Request:
  VNRecognizeTextRequest tells Vision to look for text in the image.
  Inside the request's completion handler, we extract the recognized lines from the observations.

• Extract Top Candidates:
  Each observation may have multiple text guesses. We take the top one (topCandidates(1)) to get the most confident match.

• Tune the Recognition Settings:

  • recognitionLevel = .accurate ensures high-quality OCR.

  • usesLanguageCorrection = true helps with fixing common OCR mistakes.

  • recognitionLanguages = ["fr-FR"] limits recognition to French, which improves accuracy if your receipts are in French. (Change this if needed.)

• Run the OCR Asynchronously:
  We use VNImageRequestHandler to perform the request in a background thread so the UI stays responsive.

After extracting text from the receipt, not all lines are useful—some are prices, quantities, column headers, or unrelated noise. To clean this up, we use a simple text filtering function:

What this does:

Keeps only likely product names, helping improve accuracy before classification.

Note: The filtering logic in isValidProductLine(_:) is tailored to receipts in French and specific formatting patterns.
Depending on the language and format of your receipts, you will need to adapt the filtering rules such as stop words, regular expressions, and patterns to accurately remove irrelevant lines and keep meaningful product data.

Now that we have done Parsing and Structuring Extracted Data, the next step is to classify each item into categories. We do this by passing the cleaned texts to the function: func classifyItems(from texts: [String]) -> [(text: String, category: String)]

How it works:

• We load the CreateML-generated model (MyTextClassifierFrench) which was trained on your dataset of receipt items and their categories.

• For each cleaned text line, we use the model’s prediction(text:) method to get a category label.

• If prediction fails for any reason, the label defaults to "INCONNU" (French for “unknown”).

• Finally, we return a list of tuples containing the original text and its assigned category.

This simple function is the core of the smart receipt scanner’s ability to automatically categorize items, making it easy to present structured and meaningful results to users.

App Preview: What It Looks Like in Action

Here’s a glimpse of how the app looks when scanning and categorizing receipt items:

We’ve built a working receipt scanner that uses OCR and a custom ML model all powered by on-device machine learning without diving into complex code or overwhelming theory. The goal was to show that building something smart and practical doesn’t have to be intimidating. From scanning receipts to automatically organizing them by category, you now have a solid, flexible foundation that you can build on and expand however you like. Keep building, keep exploring!