Building an AI-powered MRI Report Generation App with Streamlit and YOLOv8

In today's fast-paced medical environment, efficiency is key. With advancements in AI, we're now able to streamline complex tasks, making them faster and more accurate. One such task is the analysis and reporting of MRI images. In this post, I'll walk you through the process of building a Streamlit application that segments tumors in MRI images and generates comprehensive PDF reports using AI.

Why This Project?

Medical imaging is a critical part of diagnosing and treating conditions like brain tumors. Traditionally, radiologists manually analyze MRI scans, which can be time-consuming and prone to human error. By leveraging AI, we can assist radiologists in generating detailed reports more efficiently, allowing them to focus on patient care.

Project Overview

This project is a Streamlit application that:

• Accepts MRI images as input.

• Uses the YOLOv8 model to detect and segment tumors.

• Integrates Google Gemini AI to generate detailed medical reports based on the MRI analysis.

• Outputs a PDF report that includes the segmented image and AI-generated findings.

Key Features:

1. Upload and Process MRI Images: Users can upload MRI images, which the app then processes to detect and segment potential tumors.

2. AI-Generated Reports: Google Gemini AI analyzes the segmented MRI images and generates a comprehensive report, including the tumor's size, location, and potential diagnosis.

3. PDF Report Download: The application formats the AI-generated report into a professional PDF that users can download and share.

How It Works

1. Image Upload and Preprocessing

The user uploads an MRI image through the Streamlit interface. The image is then preprocessed to ensure it is in the correct format (RGB) for analysis by the YOLOv8 model.

uploaded_file = st.file_uploader("Upload MRI Image with Mask Overlay", type=["jpg", "jpeg", "png"])

2. Tumor Segmentation with YOLOv8

Once the image is uploaded, the YOLOv8 model is used to detect and segment the tumor in the MRI image. YOLOv8, known for its speed and accuracy, is ideal for real-time image processing tasks like this.

from yolov8 import YOLO

model = YOLO('yolov8-seg.pt')  # Load YOLOv8 model
results = model(image)

3. AI-Powered Report Generation

With the segmented image, the application sends the data to Google Gemini AI. The AI generates a detailed medical report, analyzing the segmented tumor and providing insights like tumor size, location, and possible diagnoses.

# Example report generation process
report_text = generate_report_with_gemini(image_data)

4. PDF Report Creation and Download

The generated report is then formatted into a PDF document. The PDF includes both the segmented MRI image and the AI-generated findings, providing a complete, shareable document that healthcare professionals can use.

pdf_filename = create_pdf_report_with_image(patient_name, report_text, image_path)
st.download_button(label="Download PDF Report", data=pdf_file, file_name=pdf_filename)

Integration with Google Gemini

One of the standout features of this project is the integration with Google Gemini. By leveraging this powerful AI, we can provide more than just basic image analysis. Google Gemini offers detailed, context-aware insights that elevate the quality of the report, making it a valuable tool for medical professionals.

Setting Up Google Gemini

To integrate Google Gemini, you need to:

1. Obtain an API key from Google.

2. Set up the environment variables in your project to securely access the API.

3. Use the API to send segmented image data and receive detailed analysis.

This integration allows the application to provide more than just visual data—it's capable of generating nuanced, text-based insights that enhance the overall report.

Challenges and Learnings

Markdown Parsing in PDFs

Initially, I attempted to implement markdown-like syntax (e.g., **bold** and ## headings) in the generated PDF reports. However, due to limitations in the markdown parsing and rendering within the FPDF library, this feature had to be excluded. This was a valuable lesson in understanding the limitations of certain libraries and the importance of testing early and often.

Streamlit’s Simplicity

One of the major highlights of this project was working with Streamlit. The framework’s simplicity and flexibility made it easy to develop an interactive, user-friendly interface. Its real-time updates and ease of deployment are perfect for building and sharing AI-powered applications.

Conclusion

This project demonstrates how AI can transform the way we analyze and report on medical images. By combining the power of YOLOv8 for image segmentation with the advanced analytical capabilities of Google Gemini, we've built a tool that can significantly enhance the efficiency and accuracy of MRI report generation.

Whether you're a radiologist looking to streamline your workflow or a developer interested in the intersection of AI and healthcare, this project provides a solid foundation for further exploration.

Interested in the Code?

Check out the GitHub repository for the complete code and setup instructions. Contributions and feedback are always welcome!

What’s Next?

Future improvements could include:

• Advanced Image Preprocessing: To handle different image modalities and improve segmentation accuracy.

• Enhanced Report Customization: Allowing users to add more custom fields and insights into the report.

• Integration with Other AI Models: Exploring the use of different AI models for comparative analysis and improving the accuracy of diagnoses.

Stay tuned for more updates and improvements!