Decoding Doctor’s Handwriting: How I Built SmartRx to Read & Translate Prescriptions with AI

Ever squinted at a doctor’s handwriting, trying to decode what to buy from the pharmacy? 😮‍💨

You’re not alone……. medical prescriptions are famously hard to read, and even harder to understand if they’re packed with technical jargon or written in a language you’re not comfortable with.

SmartRx is my tiny attempt to fix that. It’s a simple AI-powered tool that reads handwritten or printed prescriptions, extracts the key details, summarizes them in plain language, and even translates them to your local language, all in a few seconds, right from your browser.

Whether you’re a patient, a caregiver, or a busy pharmacist, SmartRx aims to turn messy scribbles into clear, shareable instructions, and bring a little more peace of mind when dealing with your health.

What Problem Does SmartRx “exactly” Solve? 👀

If you’ve ever stood at a pharmacy counter, second-guessing what’s scribbled on that piece of paper, you know the struggle:

• Unreadable handwriting: Many prescriptions are handwritten in a rush, and can be tough to decode, even for pharmacists.

• Medical jargon: Dosages, frequencies, and medicine names can confuse patients and caregivers who aren’t from a medical background.

• Language barriers: Not everyone reads or speaks the language the prescription is written in, especially in multilingual places like India.

• No easy way to share: Sharing instructions with family members or caretakers often means rewriting or explaining details again and again.

SmartRx tackles all these by turning messy notes into clear, simple summaries, plus translations, so anyone can understand and share them easily.

How SmartRx Works ? 🧠

I kept SmartRx as simple (and friendly) as possible……. here’s what happens behind the scenes:

1. Snap & Upload: Take a quick photo of your prescription or pharmacy bill and upload it on the web app.

2. AI Steps In: Behind the scenes, GPT-4o reads the image, picks out details like patient name, doctor’s info, date, and the full list of medicines with dosage instructions.

3. No Medical Jargon: It cleans up the extracted info and turns it into a short, plain-English summary, no complicated words, just what you really need to know.

4. Speak Your Language: Want it in your local language? SmartRx can translate the summary into Indian or global languages with a single click. (50+ languages ✌️)

5. Share & Save: Download the text, or share it straight to WhatsApp, no more sending blurry photos to confused family members.

   

Behind the Scenes: How I Built SmartRx ⛓️

Building SmartRx was a fun mix of tinkering with AI, keeping things simple for users, and gluing everything together with Streamlit.

Here’s the gist of how it works under the hood:

• Image to Text: When you upload a photo, it’s converted to base64 and sent to GPT-4o. A carefully crafted prompt tells the model exactly what to extract : patient details, doctor’s name, date, medicine list, dosage, and more.

    def image_to_base64(image_bytes):
        """Convert image bytes to base64 string."""
        return base64.b64encode(image_bytes).decode("utf-8")
  
    def extract_prescription(image_bytes):
        """Send image + prompt to GPT-4o and return structured JSON content."""
        base64_image = image_to_base64(image_bytes)
  

• Clean JSON Output: GPT returns the prescription details in a structured JSON format. This makes it easy to parse and display without messy text.

• Summarize & Translate: The same model creates a short summary in simple language. If you pick a local language, GPT translates it for you right away. And again, it works with 50+ languages across the world :)

    def translate_summary(text, target_language):
        prompt = f"""
    Translate the following text into {target_language}:
  
    {text}
        """
        response = client.chat.completions.create(
            model=deployment_name,
            messages=[
                {"role": "user", "content": prompt}
            ],
            temperature=0.3
        )
        return response.choices[0].message.content.strip()
  

• Streamlit UI: I built the whole user interface with Streamlit it’s clean, fast, and easy for anyone to use, no technical skills needed.

• WhatsApp Automation: For fun (and practicality!), I plugged in pywhatkit so you can send the final summary directly to a WhatsApp number…….. no more retyping.

    def send_whatsapp_message(receiver_number: str, message: str):
        """
        Sends a WhatsApp message using pywhatkit.
        Requires WhatsApp Web login.
        """
        try:
            print(f"Sending message to {receiver_number}...")
            pywhatkit.sendwhatmsg_instantly(receiver_number, message, wait_time=10, tab_close=True)
            time.sleep(5)  
            return True
        except Exception as e:
            print(f" Failed to send message: {e}")
            return False
  

• Safe & Lightweight: No heavy backend or complicated hosting, just environment variables for config (dotenv), and a few Python tools (pandas, json, re) to handle the data neatly.

In short: SmartRx is basically a good and smart AI brain, and a simple UI wrapped together……. so that the tech stays hidden, and you just get clear answers.

For detailed code, you can checkout the whole repository here!!

A Few Snapshots of SmartRx 📸

What’s Next for SmartRx? 🌱

SmartRx is just getting started and of course there’s so much more it can do! I’d love to add:

• 📄 Support for PDF uploads and multi-page prescriptions

• 🔍 Better OCR fallback for messy handwriting

• 🗂️ User accounts to save past prescriptions securely

• 🔊 Voice summaries……. super handy for elderly users

• 🔒 Stronger privacy and secure storage (with user consent, of course)

If you’ve got ideas or want to build on this with me, I’d genuinely love to hear from you!

Here’s the Github repo of this project : SmartRx

Thanks for reading, and If you’d like to chat, share ideas, or improve it together, my inbox is always open :)(Here’s my Linkedin)