20 Requests concepts with Before-and-After Examples

1. Basic HTTP GET Request 🌐

Boilerplate Code:

import requests

response = requests.get('https://example.com')
print(response.content)

Use Case: Retrieve data from a website via a simple GET request.

Goal: Use requests to fetch content from a webpage. 🎯

Before Example:

# Manually navigating the webpage to copy the content:
# Open browser, go to https://example.com, right-click and inspect to copy content.

After Example:

# Automating the process to fetch content using Python:
response = requests.get('https://jsonplaceholder.typicode.com/todos/1')
print(response.json())

---

2. Handling Query Parameters (GET Request with Params) 🔍

Boilerplate Code:

requests.get(url, params={'key': 'value'})

Use Case: Send GET requests with parameters, like filtering or searching data.

Goal: Use query parameters to customize requests to an API or website. 🎯

Before Example:

# Searching manually via a website:
# Open the website, enter the search query, and manually browse through filtered results.

After Example:

# Using Python to send GET requests with query parameters:
response = requests.get('https://jsonplaceholder.typicode.com/todos', params={'userId': 1})
print(response.json())

---

3. Handling POST Requests 📝

Boilerplate Code:

requests.post(url, data={'key': 'value'})

Use Case: Send data to a server using POST, like submitting forms.

Goal: Use POST to send data to APIs or websites. 🎯

Before Example:

# Submitting form data manually via a browser:
# Navigate to a website, fill out the form, and click the submit button.

After Example:

# Automating the form submission with Python POST request:
response = requests.post('https://jsonplaceholder.typicode.com/posts', data={'title': 'foo', 'body': 'bar', 'userId': 1})
print(response.json())

---

4. Custom Headers in Requests 🏷️

Boilerplate Code:

requests.get(url, headers={'Authorization': 'Bearer token'})

Use Case: Add custom headers to requests, like authentication tokens.

Goal: Use headers to authenticate or send custom metadata in requests. 🎯

Before Example:

# Manually entering an authentication token:
# Open the website, enter token in a form or via developer tools, and access restricted content.

After Example:

# Automating the process with Python and custom headers:
headers = {'Authorization': 'Bearer your_token_here'}
response = requests.get('https://jsonplaceholder.typicode.com/posts', headers=headers)
print(response.status_code)

---

Here are examples 5-9 following your preferred format with before and after:

---

5. Sending JSON Data with POST Requests 📦

Boilerplate Code:

requests.post(url, json={'key': 'value'})

Use Case: Send data in JSON format to an API.

Goal: Use POST with JSON data to communicate with APIs. 🎯

Before Example:

# Manually entering data into a form on a website:
# Navigate to the form, fill it out, and click submit.

After Example:

# Automating data submission using a Python POST request with JSON:
response = requests.post('https://jsonplaceholder.typicode.com/posts', json={'title': 'foo', 'body': 'bar', 'userId': 1})
print(response.json())

---

6. Handling Response Status Codes 📊

Boilerplate Code:

response.status_code

Use Case: Check if a request was successful by inspecting status codes.

Goal: Monitor and handle different HTTP response codes. 🎯

Before Example:

# Manually checking if a webpage or API is accessible:
# Open browser, attempt to access a page, and check if it loads.

After Example:

# Automatically checking response codes using Python:
response = requests.get('https://jsonplaceholder.typicode.com/posts/1')
if response.status_code == 200:
    print("Success!")
else:
    print(f"Failed with status code: {response.status_code}")

---

7. Timeout Handling in Requests ⏳

Boilerplate Code:

requests.get(url, timeout=5)

Use Case: Prevent long-running requests by setting time limits.

Goal: Set timeouts for requests to avoid hanging operations. 🎯

Before Example:

# Manually refresh or cancel a browser tab if a webpage takes too long to load.

After Example:

# Automatically handle timeouts with Python requests:
response = requests.get('https://jsonplaceholder.typicode.com/todos/1', timeout=3)
print(response.content)

---

8. Handling Redirects 🔄

Boilerplate Code:

requests.get(url, allow_redirects=False)

Use Case: Control how redirects are handled when a URL points to a new location.

Goal: Enable or disable automatic following of redirects. 🎯

Before Example:

# Manually follow browser redirects when you're taken to a new page.

After Example:

# Disable redirects in a Python request:
response = requests.get('https://jsonplaceholder.typicode.com/todos/1', allow_redirects=False)
print(response.status_code)

---

9. File Uploads via POST Requests 📁

Boilerplate Code:

requests.post(url, files={'file': open('file.txt', 'rb')})

Use Case: Upload files to a server using POST requests.

Goal: Automate file uploads via HTTP requests. 🎯

Before Example:

# Manually upload files via a web form: 
# Navigate to the upload page, choose a file, and click submit.

After Example:

# Automating file uploads with Python:
files = {'file': open('example.txt', 'rb')}
response = requests.post('https://jsonplaceholder.typicode.com/posts', files=files)
print(response.status_code)

---

10. Session Objects for Persistent Connections 🌐

Boilerplate Code:

session = requests.Session()

Use Case: Maintain persistent connections across multiple requests, preserving cookies and headers.

Goal: Use session objects for efficient, stateful communication with servers. 🎯

Before Example:

# Manually re-authenticating or re-sending session info for every new request in the browser.

After Example:

# Using Python requests with session object:
session = requests.Session()
session.get('https://jsonplaceholder.typicode.com/posts/1')
response = session.get('https://jsonplaceholder.typicode.com/todos/1')
print(response.json())  # Persistent connection

---

11. Cookie Handling 🍪

Boilerplate Code:

requests.get(url, cookies={'session_id': '12345'})

Use Case: Manage cookies in requests, either sending or receiving cookies.

Goal: Handle cookies automatically with requests, useful for login sessions or tracking. 🎯

Before Example:

# Manually inspecting cookies or using browser developer tools to check and set cookies for login sessions.

After Example:

# Using Python requests to send cookies with a request:
cookies = {'session_id': 'abc123'}
response = requests.get('https://jsonplaceholder.typicode.com/todos', cookies=cookies)
print(response.cookies)

---

12. Proxies in Requests 🕵️‍♂️

Boilerplate Code:

requests.get(url, proxies={'http': 'http://proxy.com:8080'})

Use Case: Route requests through a proxy server for security or access control.

Goal: Use proxies for requests, masking your IP or gaining access to restricted websites. 🎯

Before Example:

# Manually setting up proxy settings in a browser or using VPN to route traffic.

After Example:

# Using a Python proxy to route requests:
proxies = {'http': 'http://proxy.com:8080'}
response = requests.get('https://jsonplaceholder.typicode.com/todos', proxies=proxies)
print(response.status_code)

---

13. Streaming Requests (Downloading Large Files) 🚀

Boilerplate Code:

requests.get(url, stream=True)

Use Case: Download large files or data streams without loading them entirely into memory.

Goal: Stream data in chunks to handle large files efficiently. 🎯

Before Example:

# Manually downloading large files in a browser, which can cause memory issues.

After Example:

# Using Python to download large files in chunks:
response = requests.get('https://jsonplaceholder.typicode.com/photos', stream=True)
with open('photos.json', 'wb') as f:
    for chunk in response.iter_content(chunk_size=1024):
        f.write(chunk)

---

14. Error Handling with raise_for_status() ❗

Boilerplate Code:

response.raise_for_status()

Use Case: Raise exceptions for HTTP errors (e.g., 404 or 500) automatically.

Goal: Simplify error handling by raising an exception when requests fail. 🎯

Before Example:

# Manually checking status codes after each browser request, or encountering errors without a clear indication.

After Example:

# Automatically handle HTTP errors using raise_for_status:
response = requests.get('https://jsonplaceholder.typicode.com/invalid-url')
try:
    response.raise_for_status()
except requests.exceptions.HTTPError as e:
    print(f"HTTP Error: {e}")

---

15. Session Cookies (Persistent Cookies) 🍪

Boilerplate Code:

session = requests.Session()
session.cookies.update({'session_id': '12345'})

Use Case: Maintain session cookies across multiple requests for persistent login or data tracking.

Goal: Use session cookies to handle authentication across multiple requests. 🎯

Before Example:

# Manually re-authenticating for every new page request in a browser.

After Example:

# Using Python to maintain session cookies across multiple requests:
session = requests.Session()
session.cookies.update({'session_id': 'abc123'})
response = session.get('https://jsonplaceholder.typicode.com/todos')
print(response.json())

---

16. Request Hooks (Modify Requests and Responses) 🎣

Boilerplate Code:

requests.get(url, hooks={'response': hook_function})

Use Case: Modify requests or responses using hooks, such as logging requests or altering response data.

Goal: Customize requests and responses by using hooks for pre/post-processing. 🎯

Before Example:

# Manually logging requests or modifying responses using developer tools in the browser.

After Example:

# Using Python hooks to modify the response:
def log_request(r, *args, **kwargs):
    print(f"Request made to {r.url}")

response = requests.get('https://jsonplaceholder.typicode.com/todos', hooks={'response': log_request})

---

17. File Downloads with Progress Bar 📊

Boilerplate Code:

requests.get(url, stream=True)

Use Case: Download large files while displaying a progress bar for better user experience.

Goal: Track the progress of large file downloads. 🎯

Before Example:

# Manually downloading large files, with no indication of download progress.

After Example:

# Using Python to download files with a progress bar:
from tqdm import tqdm

response = requests.get('https://jsonplaceholder.typicode.com/photos', stream=True)
total_size = int(response.headers.get('content-length', 0))

with open('photos.json', 'wb') as file, tqdm(
    desc='Downloading', total=total_size, unit='B', unit_scale=True
) as bar:
    for data in response.iter_content(chunk_size=1024):
        file.write(data)
        bar.update(len(data))

---

18. Basic Authentication in Requests 🔑

Boilerplate Code:

requests.get(url, auth=('user', 'pass'))

Use Case: Access a server or API that requires basic authentication.

Goal: Send credentials using Basic Authentication to access restricted resources. 🎯

Before Example:

# Manually entering credentials on a website to log in and access content.

After Example:

# Using Python requests to handle basic authentication:
response = requests.get('https://jsonplaceholder.typicode.com/posts', auth=('username', 'password'))
print(response.status_code)

---

19. OAuth 2.0 Token Authorization 🔐

Boilerplate Code:

requests.get(url, headers={'Authorization': 'Bearer token'})

Use Case: Authenticate and access APIs that use OAuth 2.0 tokens for authorization.

Goal: Use a Bearer token to authenticate requests and access secured API resources. 🎯

Before Example:

# Manually providing OAuth tokens via form inputs or browser extensions.

After Example:

# Using Python requests to access protected resources with OAuth 2.0 token:
headers = {'Authorization': 'Bearer YOUR_ACCESS_TOKEN'}
response = requests.get('https://jsonplaceholder.typicode.com/posts', headers=headers)
print(response.json())

---

20. Lazy Loading (Deferred Load) 💤

Boilerplate Code:

session.query(User).all()  # No posts loaded yet
for user in users:
    print(user.posts)  # Posts are loaded when accessed

Use Case: Load related data only when it's accessed.

Goal: Use lazy loading to defer loading related data until it is needed, optimizing memory and performance. 🎯

Before Example:

# Manually loading all related records at once, regardless of whether they're needed.

After Example:

# Using SQLAlchemy lazy loading to optimize performance:
users = session.query(User).all()  # No posts loaded yet
for user in users:
    print(user.posts)  # Posts loaded only when accessed