Understanding cURL: A Command Line Tool for transferring data with URLs

In today's digital age, billions of API requests are made globally over the internet, seamlessly integrating technology into our daily lives. We interact with these APIs through various devices such as smartphones, smartwatches, laptops, and even televisions, which enhance our connectivity and access to information. In this article, I will delve into my personal journey of learning cURL, a powerful command line tool, sharing insights and practical examples of how I utilized cURL to effectively test and debug APIs.

What is cURL?

cURL, short for "client URL," is an open-source command line tool that facilitates data transfer over the internet using various protocols. It enables users to send and receive data from a server by specifying the endpoint and options directly through the command line. While tools like Postman, Insomnia, and Hoppscotch offer graphical user interfaces for making API requests, cURL provides a powerful alternative for performing these tasks directly from the terminal on Windows, Linux, and MacOS systems.

Why cURL?

• It’s one of the simplest ways to make API requests.

• It comes pre-installed in MacOS, Windows, Linux operating systems, allowing it to run on any OS without any installation.

• It is backward-compatible, stable, and offers around 230 command-line options.

• It is considered a Swiss-knife for internet transfer as it supports multiple protocols like HTTP, HTTPS, FTP, FILE, FTPS, LDAP, LDAPS, SMTP, SMTPS, TELNET, WSS and others.

• Debugging and testing web applications is easy as it offers verbose output-option that shows the detailed information of the request and response, headers, timings, and status codes.

• cURL supports adding custom headers, using cookies, and managing sessions, which are essential for testing APIs, debugging, and interacting with web services that need authentication or session persistence.

• It can be used in scripts and integrated with automated workflows.

• Since it works in the headless mode, it is incredibly lightweight and efficient, especially for server-side tasks and data transfers in background processes.

curl syntax

The basic syntax for curl is

curl [options] [url]

• curl is the command to tell the terminal that you are making a curl request.

• options or flags tell curl to perform specific actions and start with one or two dashes. Many options need an additional value next to them. If the provided text does not start with a dash, it is assumed to be a URL.

• url is the location where you want to send or retrieve data from.

Now that we have a basic understanding of the cURL syntax, let's explore how these commands are used in practice to perform various types of HTTP requests.

In this article, we will use dummy APIs for the demonstration.

curl commands

As curl supports multiple protocols, I am going to discuss the HTTP and HTTPs protocols.

HTTP GET

curl [url] - very basic command of curl that makes a HTTP GET request by default and fetches the content of the website. When you enter this command, HTML of the page will be shown on the terminal.

curl 'http://example.com'

Options can be provided in short (minus symbol and a single letter immediately following it) or long (two minuses, and then the name).

curl [-i or --include] [url] - curl with -i option will give headers and body in the response.

curl -i 'http://example.com'

HTTP HEAD

curl [-I or --head] [url] - curl with -I (capital i) option will fetch only the headers. This is basically a HTTP HEAD request.

curl -I 'http://example.com'

Verbose

curl [-v or --verbose] [url]: -v flag will tell curl to run in the verbose mode, makes the operation more talkative.

curl -v 'https://api.restful-api.dev/objects/7'

Silent

Silent is just the opposite of verbose. With the -s (or --silent) option you make curl switch off the progress meter and not output any error messages for when errors occur. It gets mute. It still outputs the downloaded data you ask it to.

curl -s 'https://api.restful-api.dev/objects/7'

Redirection

To handle a situation where an object is expected at a given location on the server but is actually located elsewhere, you can use the -L or --location option with cURL. This option tells cURL to follow any redirects that the server issues. For example, if you want to follow the redirect from www.twitter.com to www.x.com, you would use the following command:

This command will automatically follow the redirect to the new location.

curl -L 'http://www.twitter.com'

k flag

To ignore the SSL certificate check, you can use the -k or --insecure option. This option tells cURL to bypass the SSL certificate verification process, allowing you to connect to a website even if the SSL certificate is invalid or self-signed.

Without using -k or --insecure option

curl https://self-signed.badssl.com

With -k option

curl -k https://self-signed.badssl.com

Writing output in file

curl [-o or --output] [url] - The -o or --output option in cURL is used to write the output to a file. When using this option, you specify the filename where you want the output to be saved. The order of the URL and the -o option does not matter, but the first -o is associated with the first URL.

curl 'https://example.com' -o output.txt

curl [-O or --remote-name or --remote-name-all] [url] - The -O or --remote-name option in cURL is used to save the file with the same name as it is provided on the server. This option does not require you to specify a filename; instead, it automatically uses the name of the file as it appears in the URL.

curl -O 'http://example.com/index.html'

URL Globbing

To fetch data from a range of URLs where only a small portion changes between requests, you can use cURL's URL globbing feature. This allows you to specify a range of numbers or a set of names within the URL.

Numerical Range [N, M] - where N is the start index and goes up to M (including index M)

curl 'https://reqres.in/api/users/[1-4]'

It can accept the range with zero prefixes.

curl 'https://reqres.in/api/users/[01-03]'

To fetch every i-th record in a range [n, m] using cURL, you can use the step counter feature in URL globbing.

curl 'https://reqres.in/api/users/[1-4:2]'

Other supported formats

curl 'http://example.com/section[a-z:5].html' | | List | curl 'https://rickandmortyapi.com/api/character?name={rick,morty,summer}' | | Combinations | curl 'http://example.com/{Tom,Rick,Harry}-{50x50,100x100}.jpg'
curl 'http://example.com/checkers-[0-7]x[0-7].jpg'
curl 'http://example.com/{web,mail}-log[0-6].txt' |

Custom Headers

By default, curl sends a few standard headers. However, users can replace all headers sent by curl using the -H or --header option.

curl -H 'Host: test.example' 'http://example.com/'

curl -H 'Authorization: Bearer xxxxxxxxxxxx' 'http://example.com/'

to delete an internally generated header, just give header name followed by colon

curl -H 'User-Agent:' 'http://example.com/'

to add header with no contents, just give header name followed by semi-colon

curl -H 'User-Agent;' 'http://example.com/'

HTTP POST

Simple POST

To send form over a browser, a browser encodes URL with name and values pairs separated by & symbols. Same can be achieved by curl using -d or --data option.

curl -d 'name=charlie&city=london' 'http://example.com'

If the content is too large to send as a string, it can be sent as a file.

curl -d @datafilename 'http://example.com'

Content-Type

By default, curl uses a header Content-Type: application/x-www-form-urlencoded with -d option. In order to provide the valid content type that matches with the API specifications, use -H option. This will tell the server about what the content type is.

curl -d '{"name": "Apple MacBook Pro 16","data": {"year": 2019,"price": 1849.99,"CPU model": "Intel Core i9","Hard disk size": "1 TB"}}' -H 'Content-Type: application/json' 'https://api.restful-api.dev/objects'

Since, the payload is too large as a string, let’s pass the same using file. To pass a large payload using a file with cURL, we can use the -d or --data option followed by the @ symbol and the filename.

curl -d @payload.json -H 'Content-Type: application/json' 'https://api.restful-api.dev/objects'

Multipart formposts

A multipart formpost is an HTTP POST request where the request body is formatted into parts separated by MIME boundaries, used when an HTML form is submitted with enctype set to multipart/form-data.

With curl, use the -F (or --form) flag for each multipart, adding one -F for every input field you want to send in the form.

curl -F person=anonymous -F secret=@file.txt http://example.com/submit.cgi

HTTP PUT

The difference between PUT and POST is that POST sends data to a remote resource, while PUT updates the resource with a new version.

command for HTTP PUT request is similar to POST, however we need to pass request method PUT explicitly using -X or --request option.

curl -d @payload.json -X PUT -H 'Content-Type: application/json' 'https://api.restful-api.dev/objects/ff808181932badb601933edeb61f2ebd'

HTTP PATCH

The difference between PUT and PATCH is that PUT replaces an entire resource, while PATCH updates a resource partially.

To make an HTTP PATCH request using cURL, we need to specify the request method as PATCH using the -X or --request option. This is similar to how you would make a PUT request, but explicitly indicates that you want to perform a partial update on the resource.

curl -d '{"name": "AppleMacBookPro16 - Patch"}' -X PATCH -H 'Content-Type: application/json' 'https://api.restful-api.dev/objects/ff808181932badb601933edeb61f2ebd'

HTTP DELETE

To delete a resource using cURL, we need to specify the request method as DELETE using the -X or --request option.

curl -X DELETE 'https://api.restful-api.dev/objects/ff808181932badb601933edeb61f2ebd'

Cookies

cookies can be set explicitly in curl using -b or --cookie option

send cookies as string

curl -b 'SID=g.a000qAipD' 'http://reqres.in'

read cookies from file and send it

curl -b 'cookies.txt' 'http://reqres.in'

To write cookies to a file, use the -c or --cookie-jar option. Curl saves the cookies in the file after the transaction is complete.

curl -c cookies.txt 'http://reqres.in'

New cookie session. Tell curl a new cookie session starts by using -j, --junk-session-cookies

curl -j -b cookies.txt 'http://reqres.in/'

Write out and variables

The --write-out or -w option outputs text and information after a transfer is completed.

The available variables can be accessed by writing %{variable_name} in the string, and they are replaced with the correct value.

curl -s -o /dev/null -w "%{method}" http://example.com

Usage in real life projects

1. Test status code

    #! /bin/sh
   
    http_response_code=$(curl -s -o /dev/null -w "%{response_code}" http://example.com)
    if [ $http_response_code == "200" ]
    then
      echo "PASS"
    else
      echo "FAIL"
    fi
   

2. Test health

    #! /bin/sh
   
    response_id=$(curl -H 'Content-Type: application/json' http://example.com/health | jq -r '.status')
    if [ "$response_id" == "UP" ]
    then
      echo "PASS"
    else
      echo "FAIL"
    fi
   

3. Create object using POST, fetch data from response and pass it on GET

    #! /bin/sh
   
    id=$(curl -d @payload.json -H 'Content-Type: application/json' 'https://api.restful-api.dev/objects'|jq -r '.id')
    response_id=$(curl -H 'Content-Type: application/json' https://api.restful-api.dev/objects/$id | jq -r '.id')
    if [ "$id" == "$response_id" ]
    then
      echo "PASS"
    else
      echo "FAIL"
    fi
   

4. Store response headers and then validate.

    #! /bin/sh
   
    response_headers=$(curl -s -o /dev/null -w '%{header_json}' -d @payload.json -H 'Content-Type: application/json' 'https://api.restful-api.dev/objects')
    actual_content_type=$(echo $response_headers | jq -r '."content-type".[]')
    expected_content_type="application/json"
    actual_server=$(echo $response_headers | jq -r '.server.[]')
    expected_server="cloudflare"
    if [ "$expected_content_type" == "$actual_content_type" ] && [ "$actual_server" == "$expected_server" ]
    then
      echo "PASS"
    else
      echo "FAIL"
    fi
   

5. cURL can be effectively used in deployments and CI/CD pipelines to validate the health status of dependent APIs.

6. cURL can be used to download log files from remote servers for debugging purposes. By using cURL to make HTTP or FTP requests to the server where the log files are stored, we. can retrieve these files and save them locally for analysis

Reference

https://curl.se/

https://everything.curl.dev/index.html

https://youtu.be/V5vZWHP-RqU

Conclusion

In conclusion, cURL is an crutial tool for developers, testers, devops who need to interact with APIs and perform data transfers over the internet. Its versatility, ease of use, and support for a wide range of protocols make it a powerful option for testing, debugging, and automating tasks. Whether you're making simple GET requests or handling complex multipart formposts, cURL provides the flexibility and functionality needed to efficiently manage web communications. By mastering cURL, you can enhance your ability to work with APIs and streamline your workflow, making it an essential skill in today's technology-driven world.