How’s My Eating? - Part 3

Introduction

In my last post, I shared how I built a labeling interface using Rust and React to support the development of “How’s My Eating?”—an app designed to help people slow down their eating habits using kinetic data from Apple AirPods. Since then, I’ve taken another significant step forward: automating the labeling process to save time and make it more efficient.

Manually labeling data, especially identifying every mouth movement in a video, was taking up way too much time. To speed things up, I shifted to using OpenCV with Python to detect key points of my mouth and jaw. Now, I’m using a Tauri application that combines a React front end with Python executables to handle much of the heavy lifting.

Why Automate Labeling?

When training a machine learning model, the accuracy and efficiency of your labels are crucial. The more accurately I can label the data, the better the model will perform. However, manually labeling every instance of mouth movement—chewing, swallowing, or taking a new bite—was incredibly time-consuming. I needed a faster, more efficient way to get the job done, which led me to automation.

By using OpenCV, I can detect points around my mouth and jaw and calculate the ratios between them to find patterns that correspond to specific actions. For example, when I take a new bite of food, my hand often covers some of these points, which I can also track to differentiate between taking a bite and just chewing. This approach has made the labeling process much more manageable.

Building the Automated Labeling System with Tauri

To bring this idea to life, I created a Tauri application with a React front end that communicates with Python executables. Here’s a closer look at how it works:

• Frontend (React): The front end is built with React, providing an intuitive interface for loading videos, displaying motion data, and showing detected points in real-time. As the video plays, the React app visualizes key points on the mouth and jaw, allowing me to verify the automated labels easily.

• Backend (Python + OpenCV): The heavy lifting happens on the backend. Python scripts, executed from within the Tauri app, use OpenCV to process the video frames, detect key points, and calculate the necessary ratios to identify chewing, swallowing, or taking a new bite. The Python scripts then pass this labeled data back to the React app for visualization and adjustment if needed.

• Integration (Rust): Tauri, built using Rust, acts as the bridge between the React front end and Python executables, allowing them to communicate efficiently. This setup keeps the app lightweight and ensures the backend processes can run smoothly without overburdening the user’s device.

The Unexpected: Pac-Man

While analyzing the data, I stumbled upon something quite amusing. When I plotted certain data points in a pie chart format, it looked strikingly like Pac-Man eating! 🟡👾 As the video plays, the pie chart animates in a way that mirrors the arcade character munching away, adding a humorous touch to what was otherwise a pretty technical process. It’s a fun, unexpected visual that makes data analysis a bit more entertaining!

What’s Next?

Now that the automated labeling system is up and running, the next steps involve refining it further to make sure it captures the nuances of eating behavior as accurately as possible. This includes:

• Improving Detection Accuracy: Tweaking the OpenCV scripts to better handle edge cases, like low lighting or unusual mouth movements.

• Model Training: Using the labeled data to train a model. I’m starting with Google’s Teachable Machine for simplicity and planning to explore more robust frameworks like PyTorch or CoreML depending on the results.

If you want to follow the project’s progress or contribute to its development, check out the GitHub repository here: Hows-My-Eating

Stay tuned for more updates, and as always, thanks for reading!