How to add a robots.txt file in Open edX

What is the robots.txt file?

The robots.txt file is a standard text file that tells search engine crawlers which pages they can access, scrape, and ultimately list in their search engine results.

The file is listed on the root URL of your site, https://domain.com/robots.txt, and can easily be added to your Django project.

Be sure to use the correct casing and spelling for anything added to the robots.txt file or it will not work correctly.

A simple robots.txt file

User-agent: *
Disallow: /

This file will disallow all robots.

Create a robots.txt file

You have two options about where to create the robots.txt file in your Open edX project:

1. In your base templates of LMS at the following path, edx-platform/lms/templates/robots.txt

2. In your theme directory (if you have enabled comprehensive theming), edx-platform/themes/my-theme/lms/templates/robots.txt

Either way is correct but it is preferable to enable comprehensive theming and follow the second option.

Add a URL path to robots.txt

Go to the edx-platform/lms/urls.py file and add the following code:

from django.views.generic.base import TemplateView

# For koa or earlier releases
urlpatterns = [
    url(r'^robots.txt', TemplateView.as_view(template_name="robots.txt", content_type="text/plain")),
]

# For lilac or later releases
urlpatterns = [
    path("robots.txt", TemplateView.as_view(template_name="robots.txt", content_type="text/plain")),
]

Restart your LMS service and you can see the robots.txt file at: https://domain.com/robots.txt

References:

• https://searchfacts.com/robots-txt-allow-disallow-all/

• https://www.ordinarycoders.com/blog/article/robots-text-file-django