Developing a Reddit Gig Scraper in 24 Hours: A Practical Automation Approach

Michael EdwardMichael Edward
4 min read

Introduction

Necessity often drives innovation, and in this case, my need to efficiently track gig opportunities on Reddit led me to develop an automated solution. Manually checking subreddits daily was neither practical nor scalable, so I built a bot that could monitor relevant subreddits and notify me in real time whenever new opportunities appeared. Over the course of 24 hours, I designed and implemented a Reddit Job Monitor, an efficient and fully automated solution.

Identifying the Problem: Efficiently Tracking Gigs on Reddit

Reddit hosts a wealth of freelance and gig opportunities, but manually browsing through multiple subreddits is time-consuming given the sheer volume of posts. Posts can quickly become buried, reducing visibility and response time. A more efficient approach was necessary—one that would allow me to automatically scan subreddits for specific keywords, validate the credibility of the poster to mitigate scams, and deliver notifications to my phone in real time.

Instead of manually sifting through posts, I sought to offload the repetitive work to a script. This required designing a bot that could:

  • Monitor specific subreddits for new posts matching relevant keywords.

  • Evaluate post authors based on karma to filter out potential scams.

  • Send instant notifications via Telegram when a qualifying post was found.

  • Operate efficiently and in compliance with Reddit’s policies.

Choosing the Right Tools for the Job

My initial thought was to use Selenium, a popular tool for web automation. However, after consulting the Reddit developer community on r/redditdevs, I realized that Selenium was not the optimal choice for this task. Selenium is primarily suited for interacting with dynamic web elements, whereas my project required structured data extraction from Reddit.

Instead, I opted for PRAW (Python Reddit API Wrapper), a Python library specifically designed for interacting with Reddit’s API. This decision was driven by several factors:

  • Efficiency: PRAW directly fetches JSON data from Reddit, eliminating the need to render entire webpages.

  • Compliance: Using PRAW ensures adherence to Reddit’s API guidelines, reducing the risk of being flagged as a bot.

  • Simplicity: The library abstracts away complexities, making it easy to retrieve subreddit posts with minimal code.

  • Speed: By working directly with Reddit’s API, PRAW allows for significantly faster data retrieval compared to web scraping.

To complete the automation pipeline, I integrated Telegram Bot API to deliver real-time notifications whenever a relevant post was detected. This eliminated the need to constantly check Reddit manually, enabling instant responses to new gig opportunities.

Implementing the Reddit Job Monitor

The bot functions by continuously monitoring selected subreddits at scheduled intervals, scanning new posts for relevant keywords, filtering out low-quality or scam posts based on author karma, and sending formatted messages to my Telegram chat when a post meets the criteria.

Setting Up the Environment

To replicate this setup, the following dependencies need to be installed:

pip install -r req.txt

Sensitive credentials, such as API keys and authentication tokens, should be stored securely in a .env file:

CLIENT_ID=your_reddit_client_id
CLIENT_SECRET=your_reddit_client_secret
USER_AGENT=your_user_agent
TELEGRAM_BOT_TOKEN=your_bot_token
TELEGRAM_CHAT_ID=your_chat_id
MIN_AUTHOR_KARMA=10
SUBREDDITS=forhire, slavelabour
KEYWORDS=hiring, task, freelance, remote

Core Implementation

The following Python script demonstrates how the bot retrieves new posts, applies filtering logic, and sends notifications:

import os
import praw
import telegram
from dotenv import load_dotenv

# Load environment variables
load_dotenv()

# Initialize Reddit client
reddit = praw.Reddit(
    client_id=os.getenv("CLIENT_ID"),
    client_secret=os.getenv("CLIENT_SECRET"),
    user_agent=os.getenv("USER_AGENT")
)

# Initialize Telegram bot
bot = telegram.Bot(token=os.getenv("TELEGRAM_BOT_TOKEN"))
chat_id = os.getenv("TELEGRAM_CHAT_ID")

# Function to monitor subreddits
def check_subreddits():
    subreddits = os.getenv("SUBREDDITS").split(",")
    keywords = os.getenv("KEYWORDS").split(",")
    min_karma = int(os.getenv("MIN_AUTHOR_KARMA"))

    for subreddit in subreddits:
        for post in reddit.subreddit(subreddit).new(limit=10):
            if any(keyword in post.title.lower() for keyword in keywords):
                if post.author.comment_karma >= min_karma:
                    message = f"New Gig Found: {post.title}\n{post.url}"
                    bot.send_message(chat_id=chat_id, text=message)

Automating Execution

To ensure continuous operation, the script should run at regular intervals. This can be achieved using cron (Linux/macOS) or Task Scheduler (Windows). For example, on Linux, the following cron job executes the script every 15 minutes:

*/15 * * * * /usr/bin/python3 /path/to/main.py

Key Takeaways

This project highlighted several important lessons in software development:

  • Rapid prototyping is powerful: It took just 24 hours to go from concept to a working solution.

  • Choosing the right tools is critical: Switching from Selenium to PRAW drastically improved efficiency and maintainability.

  • Automation can significantly enhance productivity: By eliminating the need for manual job searches, I can now respond to new opportunities instantly.

  • Security best practices matter: Proper handling of API credentials (e.g., using environment variables) prevents accidental exposure.

Potential Enhancements

While the bot is functional, there are several ways to extend its capabilities:

  • Advanced filtering: Incorporating NLP techniques to detect scam posts more accurately.

  • Database integration: Storing job posts to prevent duplicate alerts and enable historical analysis.

  • Multi-platform notifications: Expanding alerts to Discord, email, or SMS for broader accessibility.

  • User customization: Allowing users to dynamically specify subreddits and keywords through a web interface.

Conclusion

This project started as a simple automation task but quickly evolved into a practical and efficient tool. By leveraging PRAW and Telegram’s API, I was able to build a system that not only saves time but also enhances my ability to secure freelance opportunities. If you’re performing repetitive tasks manually, consider automating them—small investments in automation can lead to significant efficiency gains.

5
Subscribe to my newsletter

Read articles from Michael Edward directly inside your inbox. Subscribe to the newsletter, and don't miss out.

Written by

Michael Edward
Michael Edward

Software Developer