Cheatsheet & Examples: wget

Download a single file
Example Usage:
wget http://example.com/file.txt
What it does: Downloads the file located at the specified URL and saves it in the current directory with the original filename.
Command-line Arguments Explained:
http://example.com/file.txt
: The URL of the file to be downloaded.
Save a file with a custom name
Example Usage:
wget -O custom_name.txt http://example.com/actual_file.txt
What it does: Downloads the file from the specified URL and saves it under the custom name provided.
Command-line Arguments Explained:
-O
: Specifies the output filename for the downloaded file.custom_name.txt
: The desired filename for the saved file.http://example.com/actual_file.txt
: The URL of the file to be downloaded.
Recursively download a website
Example Usage:
wget -r http://example.com
What it does: Downloads an entire website, including all linked pages, recursively, and saves them locally.
Command-line Arguments Explained:
-r
: Enables recursive downloading, which means it will traverse and download all pages linked to the target URL.http://example.com
: The URL of the website to be downloaded.
Download multiple files from a list
Example Usage:
wget -i file_list.txt
What it does:
Downloads all files listed in the file_list.txt
text file, one by one.
Command-line Arguments Explained:
-i
: Reads the list of URLs from the specified input file.file_list.txt
: A text file containing the URLs of the files to be downloaded.
Check if a file exists without downloading it
Example Usage:
wget --spider http://example.com/file.txt
What it does: Verifies whether the file at the URL exists and is accessible, without initiating a download.
Command-line Arguments Explained:
--spider
: Simulates checking the URL (like a "spider" bot) without downloading the file.
Download in the background
Example Usage:
wget -b http://example.com/large_file.zip
What it does: Downloads the file in the background, allowing the user to continue using the terminal.
Command-line Arguments Explained:
-b
: Runswget
in the background. Output is logged to a file.
Limit download speed
Example Usage:
wget --limit-rate=100k http://example.com/large_file.zip
What it does: Restricts the download speed to the specified value (e.g., 100 kilobytes per second).
Command-line Arguments Explained:
--limit-rate=100k
: Limits the download speed to 100 kilobytes per second.
Resume an interrupted download
Example Usage:
wget -c http://example.com/partial_file.zip
What it does: Resumes a partially downloaded file from the last point of interruption.
Command-line Arguments Explained:
-c
: Enables resuming of an interrupted download.
Download only if the file is newer
Example Usage:
wget -N http://example.com/updated_file.txt
What it does: Downloads the file only if it is newer than the local version, or if the local file does not exist.
Command-line Arguments Explained:
-N
: Enables time-stamping, checking if the remote file is newer than the local copy.
Download with authentication
Example Usage:
wget --user=username --password=secret http://example.com/protected_file.txt
What it does: Downloads a file that requires HTTP authentication credentials.
Command-line Arguments Explained:
--user
: Specifies the username for authentication.--password
: Specifies the password for authentication.
Use a specific user agent
Example Usage:
wget --user-agent="Mozilla/5.0" http://example.com
What it does: Sets a custom user agent string to mimic a browser or bypass restrictions based on user agent detection.
Command-line Arguments Explained:
--user-agent
: Defines the user agent string sent in the HTTP request header.
Download a file with a timeout
Example Usage:
wget --timeout=30 http://example.com/really_slow_file.zip
What it does: Sets a timeout for the connection, terminating the download if no response is received within the specified seconds.
Command-line Arguments Explained:
--timeout=30
: Specifies the connection timeout in seconds (e.g., 30 seconds).
Mirror a website (recursively and retain directory structure)
Example Usage:
wget --mirror http://example.com
What it does: Mirrors the website, including all subpages and maintaining the original directory structure.
Command-line Arguments Explained:
--mirror
: A shortcut for-r --no-parent --convert-links
, which recursively downloads and preserves the site's structure.
Download from a secure (HTTPS) site without certificate verification
Example Usage:
wget --no-check-certificate https://example.com/secured_file.txt
What it does: Downloads a file from an HTTPS site without verifying the SSL certificate, useful for self-signed certificates.
Command-line Arguments Explained:
--no-check-certificate
: Skips SSL certificate verification.
Download multiple files with a single command
Example Usage:
wget http://example.com/file1.txt http://example.com/file2.txt
What it does: Downloads multiple files by specifying their URLs in a single command.
Command-line Arguments Explained:
http://example.com/file1.txt http://example.com/file2.txt
: URLs of the files to be downloaded sequentially.
Output to a specific directory
Example Usage:
wget -P /path/to/dir http://example.com/file.txt
What it does: Saves the downloaded file to the specified directory instead of the current working directory.
Command-line Arguments Explained:
-P
: Specifies the directory where the file should be saved.
Download a file with a custom header
Example Usage:
wget --header="Authorization: Bearer token" http://example.com/api/data
What it does: Attaches a custom HTTP header (e.g., for authentication) to the download request.
Command-line Arguments Explained:
--header="Authorization: Bearer token"
: Adds the specified HTTP header to the request.
Specify the proxy server
Example Usage:
wget --proxy=on http://example.com/file.txt
What it does: Uses the specified proxy server for the download, with proxy settings read from environment variables or the default configuration.
Command-line Arguments Explained:
--proxy=on
: Enables the use of a proxy server. Proxies are typically set via environment variables likeHTTP_PROXY
orHTTPS_PROXY
.
Download files with a retry limit
Example Usage:
wget --tries=5 http://example.com/retry_file.txt
What it does: Retries the download up to the specified number of times if it fails.
Command-line Arguments Explained:
--tries=5
: Sets the maximum number of retry attempts for failed downloads.
Download files with a specific output document location
Example Usage:
wget --output-document=/path/to/save/file.txt http://example.com/remote_file.txt
What it does: Saves the downloaded file to a specific location, overriding the default filename.
Command-line Arguments Explained:
--output-document
: Specifies the exact path and filename for the saved file.
Download a file using the POST method
Example Usage:
wget --post-data="key=value" http://example.com/post_endpoint
What it does: Sends a POST request with the given data to the specified URL.
Command-line Arguments Explained:
--post-data="key=value"
: Sends the specified data as the body of the POST request.
Skip parent directories during recursion
Example Usage:
wget -r --no-parent http://example.com
What it does: Recursively downloads a website but restricts the process to the exact directory specified, not descending into parent directories.
Command-line Arguments Explained:
-r
: Enables recursive downloading.--no-parent
: Preventswget
from accessing parent directories of the specified URL.
Subscribe to my newsletter
Read articles from Hong directly inside your inbox. Subscribe to the newsletter, and don't miss out.
Written by

Hong
Hong
I am a developer from Malaysia. I work with PHP most of the time, recently I fell in love with Go. When I am not working, I will be ballroom dancing :-)