How to Detect and Track Objects in Real-Time with a MacBook Camera 📷

Niladri DasNiladri Das
12 min read

In this blog post, we will discuss an exciting project that involves real-time object detection and tracking using a MacBook camera. This project combines computer vision and deep learning techniques to create a versatile application with various practical applications in traffic management, urban planning, and public safety.

Project Overview 🎽

The primary objective of this project is to develop a system that uses computer vision to detect and identify various objects in real time from the live feed of your laptop camera. The main components of this project include:

  1. Camera Feed: Capturing video input from the MacBook's camera.

  2. Object Detection Model: A pre-trained deep learning model (e.g., YOLO, SSD, Faster R-CNN) to detect and classify objects in each frame.

  3. Processing Pipeline: Handling video frames, applying the detection model, and overlaying the results on the video feed.

  4. Display Interface: Showing the annotated video feed with detected objects and relevant information.

Technologies and Tools ⚒️

  • Programming Language: Python, commonly used for computer vision tasks.

  • Libraries: OpenCV for video processing, TensorFlow or PyTorch for deep learning models.

  • Pre-trained Models: Models available from frameworks like TensorFlow Model Zoo or PyTorch Hub.

Potential Outcomes 🤩

  • Real-time Object Detection: Identify and classify different objects in the video feed, such as cars, trucks, buses, motorcycles, pedestrians, and traffic signs.

  • Traffic Monitoring: Gather data on traffic density, flow, and patterns. Identify traffic congestion and potential bottlenecks.

  • Safety and Compliance: Detect and track pedestrians to ensure safety measures. Monitor traffic rule compliance, such as detecting vehicles running red lights or illegal parking.

  • Data Collection for Analysis: Collect and store data for further analysis, such as traffic studies or urban planning.

  • Alerts and Notifications: Generate real-time alerts for specific conditions, such as detecting emergency vehicles or accidents.

  • Environmental Monitoring: Estimate vehicle emissions and analyze environmental impact based on traffic data.

Extending the Project 📊

  • Enhanced Detection Models: Train custom models to detect specific types of vehicles or objects relevant to your application.

  • Multi-Camera Setup: Use multiple cameras for a broader view and more comprehensive monitoring.

  • Integration with Other Systems: Connect with traffic management systems or city planning tools for a more integrated approach.

  • Mobile Deployment: Adapt the project for mobile devices to provide on-the-go traffic monitoring.

Setting Up a Wireless Camera 📷

If you want to set up a wireless camera, you can still fetch its video feed for your object detection project. Here's a step-by-step guide on how to do it:

  1. Choose a Wireless Camera: Ensure it supports video streaming over the network (e.g., IP cameras, Wi-Fi cameras).

  2. Install and Configure the Camera: Follow the manufacturer's instructions to connect the camera to your Wi-Fi network. Assign a static IP address to the camera if possible to make accessing it easier.

  3. Access the Camera Stream: Most wireless cameras provide a video stream URL (RTSP, HTTP, or similar) that can be used to access the video feed.

Here's a code snippet for fetching a video stream from a wireless camera using OpenCV in Python:

import cv2

# Replace with your camera's stream URL
stream_url = 'rtsp://your_camera_ip_address:port/stream'

# Open the video stream
cap = cv2.VideoCapture(stream_url)

while True:
    ret, frame = cap.read()
    if not ret:
        break

    # Perform object detection on the frame
    #...

    # Display the resulting frame
    cv2.imshow('Frame', frame)

    # Press 'q' to exit the loop
    if cv2.waitKey(1) & 0xFF == ord('q'):
        break

# When everything done, release the capture
cap.release()
cv2.destroyAllWindows()

Manually Controlling the Camera Movement

If you want to manually control the camera movement with your laptop and zoom in on specific objects, you can do so by using a PTZ (Pan-Tilt-Zoom) camera. Here's how you can achieve this:

  1. PTZ Camera: Ensure the wireless camera you are using supports PTZ features.

  2. Control Interface: Most PTZ cameras come with software or a web interface to control movement and zoom. You can also control the camera programmatically using APIs.

Object Detection Model Selection ✅

For this project, you have two main options regarding the object detection model: using a pre-trained model or training a custom model.

Using a Pre-trained Model 💪

Advantages:

  • Time-saving: Pre-trained models are ready to use and save a significant amount of time and computational resources.

  • General Performance: Pre-trained models are usually trained on large datasets and perform well on a wide range of objects.

Disadvantages:

  • Limited Customization: Pre-trained models may not perform optimally for specific objects or environments not included in their training data.

Pre-trained models available from libraries like TensorFlow, PyTorch, and OpenCV are trained to recognize these objects. Here’s an example of how to use a pre-trained model with TensorFlow:

import tensorflow as tf
import numpy as np
import cv2

# Load a pre-trained object detection model from TensorFlow Hub
model = tf.saved_model.load("http://download.tensorflow.org/models/object_detection/ssd_mobilenet_v1_coco/saved_model")

# Load an image or video frame
image = cv2.imread('path_to_image.jpg')
input_tensor = tf.convert_to_tensor(image)
input_tensor = input_tensor[tf.newaxis, ...]

# Perform object detection
detections = model(input_tensor)

# Extract detection results
num_detections = int(detections.pop('num_detections'))
detections = {key: value[0, :num_detections].numpy() for key, value in detections.items()}
detections['num_detections'] = num_detections

# Detection classes should be ints.
detections['detection_classes'] = detections['detection_classes'].astype(np.int64)

# Visualize the detection results
for i in range(num_detections):
    box = detections['detection_boxes'][i]
    class_id = detections['detection_classes'][i]
    score = detections['detection_scores'][i]

    if score > 0.5:  # Filter out low confidence detections
        y1, x1, y2, x2 = box
        cv2.rectangle(image, (int(x1 * image.shape[1]), int(y1 * image.shape[0])),
                      (int(x2 * image.shape[1]), int(y2 * image.shape[0])), (0, 255, 0), 2)
        cv2.putText(image, str(class_id), (int(x1 * image.shape[1]), int(y1 * image.shape[0]) - 10),
                    cv2.FONT_HERSHEY_SIMPLEX, 0.9, (36, 255, 12), 2)

cv2.imshow('Object Detection', image)
cv2.waitKey(0)
cv2.destroyAllWindows()

Training a Custom Model (Find in this Blog) 🎒

Advantages:

  • Customization: You can tailor the model to detect specific objects relevant to your project.

  • Better Performance: Custom models can achieve higher accuracy for specialized tasks by training on domain-specific datasets.

Disadvantages:

  • Time and Resources: Training a custom model requires a labeled dataset, significant computational resources, and time.

  • Complexity: The process of preparing data, training the model, and fine-tuning can be complex.

Data Storage Requirements 🥴

The amount of data storage required for this project depends on several factors, including the choice of model, the size of the datasets you plan to use, and how you intend to store your data and results. Here are some considerations to help you estimate the storage requirements:

  • Pre-trained Model Size: Pre-trained models vary in size. For example, YOLOv3 is ~236 MB, YOLOv4 is ~245 MB, and SSD MobileNet V2 is ~14 MB.

  • Dataset Size: If you use a dataset for fine-tuning or evaluation, the size of the dataset can vary significantly. For example, the COCO dataset is ~25 GB, and the Pascal VOC dataset is ~2 GB.

  • Video Storage: If you plan to store video feeds or frames from the camera, the storage required will depend on the video resolution, frame rate, and compression format.

  • Intermediate Data and Results: Storing processed frames, detection results, logs, etc., will also require storage, although typically less than raw video data.

YOLO: An Overview 😎

YOLO (You Only Look Once) is a popular real-time object detection algorithm known for its speed and accuracy. YOLO treats object detection as a single regression problem, which allows it to achieve high speeds while maintaining good accuracy.

Key Features of YOLO: 🎉

  1. Speed: YOLO processes images in real-time (30-60 FPS on a high-end GPU).

  2. Accuracy: Despite its speed, YOLO maintains high detection accuracy.

  3. Unified Model: YOLO treats detection as a single regression problem, which simplifies the architecture and makes it more efficient.

  4. Generalization: YOLO’s method of looking at the entire image during training and test time helps it generalize well to unseen data.

How YOLO Works: 🎈

  1. Grid Division: YOLO divides the input image into an SxS grid.

  2. Bounding Boxes and Class Probabilities: Each grid cell predicts B bounding boxes and confidence scores for those boxes. Confidence scores reflect the probability that a bounding box contains an object and how accurate the bounding box is.

  3. Class Predictions: Each grid cell also predicts C-class probabilities for the object.

  4. Non-Maximum Suppression: YOLO applies non-maximum suppression to reduce the number of overlapping boxes.

Steps to Make a YOLO Object Detection Model: 👩‍🚀

  1. Setup Environment:

    • Install necessary libraries: TensorFlow, PyTorch, OpenCV, and YOLO-specific libraries.

    • Use a high-performance GPU if available.

  2. Acquire and Prepare Data:

    • Use datasets like COCO or Pascal VOC, or collect and annotate your custom dataset.

    • Ensure data is in the correct format (e.g., YOLO format or COCO JSON format).

  3. Train the Model:

    • Choose a YOLO version (e.g., YOLOv3, YOLOv4, YOLOv5).

    • Download pre-trained weights if available to fine-tune on your dataset.

    • Configure the model architecture and hyperparameters.

    • Train the model using a deep learning framework like PyTorch or TensorFlow.

  4. Evaluate and Test:

    • Evaluate the trained model on validation and test datasets.

    • Fine-tune the model based on performance metrics like mAP (mean Average Precision).

  5. Deploy the Model:

    • Use the trained model to detect objects in real time from images or video streams.

    • Implement post-processing steps like non-maximum suppression.

Skills Required for Developing a YOLO Object Detection Model 🌊

  • Programming Skills: Proficiency in Python is essential. Familiarity with deep learning frameworks like TensorFlow or PyTorch.

  • Understanding of Deep Learning: Knowledge of convolutional neural networks (CNNs) and how they work.

  • Data Handling: Skills in data collection, annotation, and preprocessing. Familiarity with datasets like COCO and Pascal VOC.

  • Model Training and Evaluation: Ability to configure and train neural networks. Skills in evaluating model performance using metrics like mAP, precision, and recall.

  • Deployment: Experience in deploying machine learning models for real-time applications. Knowledge of using GPUs for inference to ensure real-time performance.

Example: Training YOLOv5 Using PyTorch 🤘

Here’s a simplified example of training a YOLOv5 model using the PyTorch framework:

  1. Setup Environment:
# Clone YOLOv5 repository
git clone https://github.com/ultralytics/yolov5
cd yolov5

# Install dependencies
pip install -r requirements.txt
  1. Prepare Data:

  • Organize your dataset in the YOLO format.

  • Create a dataset configuration file (e.g., data.yaml) specifying the paths and class names.

  1. Train the Model:
# Train YOLOv5
python train.py --img 640 --batch 16 --epochs 50 --data data.yaml --cfg yolov5s.yaml --weights yolov5s.pt
  1. Evaluate and Test:
# Evaluate the model
python val.py --weights runs/train/exp/weights/best.pt --data data.yaml

Loading the Model 🌊

import torch
from PIL import Image
import cv2

# Load model
model = torch.hub.load('ultralytics/yolov5', 'custom', path='runs/train/exp/weights/best.pt')

  • torch.hub.load('ultralytics/yolov5', 'custom', path='runs/train/exp/weights/best.pt') loads the YOLOv5 model with your custom-trained weights.

  • Replace 'runs/train/exp/weights/best.pt' with the actual path to your trained model's weights.

Inference on an Image

# Load image
img = Image.open('path_to_image.jpg')

# Inference
results = model(img)

# Display results
results.show()

  • Image.open('path_to_image.jpg') loads the image from the specified path.

  • model(img) performs inference on the image.

  • results.show() displays the inference results, including bounding boxes and labels.

Inference on a Video Stream

# Or for video stream
cap = cv2.VideoCapture('path_to_video.mp4')  # or use 0 for webcam
while cap.isOpened():
    ret, frame = cap.read()
    if not ret:
        break

    # Inference
    results = model(frame)

    # Display results
    results.render()
    cv2.imshow('YOLOv5 Inference', frame)
    if cv2.waitKey(1) & 0xFF == ord('q'):
        break

cap.release()
cv2.destroyAllWindows()

  • cap = cv2.VideoCapture('path_to_video.mp4') opens the video file or webcam stream. Replace 'path_to_video.mp4' with 0 to use the webcam.

  • The while loop reads frames from the video stream.

  • model(frame) performs inference on each frame.

  • results.render() draws the bounding boxes and labels on the frame.

  • cv2.imshow('YOLOv5 Inference', frame) displays the frame with the inference results.

  • The loop breaks if the q key is pressed.

Additional Details

  1. Dependencies:

    • Ensure you have torch, Pillow, and opencv-python installed:

        pip install torch pillow opencv-python

  2. Custom Model Path:

    • Make sure the path to your trained weights is correct. If your training run saved the weights in a different location, update the path accordingly.

  3. Video Source:

    • For real-time inference from a webcam, use cap = cv2.VideoCapture(0).

By using this code, you can leverage your custom-trained YOLOv5 model to detect objects in both images and video streams.

Increase Speed by just looking here: Explained Already🔥

import torch
from PIL import Image
import cv2

# Load model
model = torch.hub.load('ultralytics/yolov5', 'custom', path='runs/train/exp/weights/best.pt')

# Load image
img = Image.open('path_to_image.jpg')

# Inference
results = model(img)

# Display results
results.show()

# Or for video stream
cap = cv2.VideoCapture('path_to_video.mp4')
while cap.isOpened():
    ret, frame = cap.read()
    if not ret:
        break

    # Inference
    results = model(frame)

    # Display results
    results.render()
    cv2.imshow('YOLOv5 Inference', frame)
    if cv2.waitKey(1) & 0xFF == ord('q'):
        break

cap.release()
cv2.destroyAllWindows()

Datasets for Object Detection

Pascal VOC

Pascal VOC (Visual Object Classes) is a widely used dataset for object detection and image classification tasks. The Pascal VOC dataset includes 20 object categories:

  1. Person: Person.

  2. Animals: Bird, cat, cow, dog, horse, sheep.

  3. Vehicles: Aeroplane, bicycle, boat, bus, car, motorbike, train.

  4. Indoor Objects: Bottle, chair, dining table, potted plant, sofa, TV/monitor.

  • Size: ~2 GB

  • Number of Images: ~11,000

  • Number of Objects: ~27,000

  • Number of Classes: 21

COCO

COCO (Common Objects in Context) is a large-scale object detection, segmentation, and captioning dataset. Pre-trained models on COCO can detect and track 80 object categories, including:

  1. Person: Pedestrians, cyclists, and other people.

  2. Vehicles: Cars, trucks, buses, motorcycles, bicycles.

  3. Animals: Dogs, cats, birds, horses, cows, sheep, elephants, bears, zebras, giraffes.

  4. Household Items: Chairs, couches, potted plants, beds, dining tables, toilets, TVs, laptops, mice, keyboards, remote controls, cell phones, microwaves, ovens, toasters, sinks, refrigerators, books, clocks, vases, scissors, teddy bears, hair driers, toothbrushes.

  5. Food: Apples, oranges, bananas, grapes, strawberries, sandwiches, hot dogs, pizzas, donuts, cakes.

  6. Outdoor Objects: Fire hydrants, stop signs, parking meters, benches, birds, cats, dogs, horses, sheep, cows, elephants, bears, zebras, giraffes.

  7. Miscellaneous Objects: Backpacks, umbrellas, handbags, ties, suitcases, frisbees, skis, snowboards, sports balls, kites, baseball bats, baseball gloves, skateboards, surfboards, tennis rackets, bottles, wine glasses, cups, forks, knives, spoons, bowls.

  • Size: ~25 GB

  • Number of Images: ~330,000

  • Number of Objects: ~2,500,000

  • Number of Classes: 80

Conclusion

In this blog post, we discussed a project that involves real-time object detection and tracking using a MacBook camera. We explored the potential outcomes, technologies, and tools required for this project. Additionally, we touched upon setting up a wireless camera and manually controlling the camera movement for PTZ cameras. Finally, we discussed the option of using pre-trained models or training custom models and the skills required for developing a YOLO object detection model. This project showcases your skills in computer vision and real-time processing and provides practical applications in traffic management, urban planning, and public safety.

Cheers! ❤️

Niladri Das

0
Subscribe to my newsletter

Read articles from Niladri Das directly inside your inbox. Subscribe to the newsletter, and don't miss out.

ObjectDetectionModelEvaluationComputer Sciencedata structuresalgorithmsMachine LearningDeep LearningArtificial IntelligenceData ScienceYOLOModelTraining#AlgorithmDesignneural networksConvolutionalNeuralNetworksimage processing

Written by

Niladri Das
Niladri Das