In this article, we will explore the command-line tool jq, a powerful and flexible utility for managing and parsing JSON files.

Why use jq when JSON can be managed with other desktop and online tools?

• jq is easy to install. It has no runtime dependencies; you just need to download a single binary or securely copy it onto any machine to make it work.

• jq allows you to slice both simple and complex data efficiently with minimal effort.

• jq can be used to filter, map, and transform structured data easily according to your needs.

• jq is similar to other command-line tools like sed, awk, and grep, which let you work with text.

• jq can be combined with curl to validate JSON responses and store the responses in files.

Install jq

jq is pretty easy to install. Please refer to this link to install the jq on your machine.

To verify that jq is installed successfully, run jq --version in the terminal on Mac, Linux, or Unix machines, and in PowerShell on Windows machines.

jq commands

We will start with the basic filters. jq takes the input JSON and perform actions based on the given filters.

I will demonstrate how to use jq with a static JSON file. The files, user.json and users.json, contain non-formatted JSON data.

file: user.json

{"id":1,"name":"Leanne Graham","username":"Bret","email":"Sincere@april.biz","address":{"street":"Kulas Light","suite":"Apt. 556","city":"Gwenborough","zipcode":"92998-3874","geo":{"lat":"-37.3159","lng":"81.1496"}},"phone":"1-770-736-8031 x56442","website":"hildegard.org","company":{"name":"Romaguera-Crona","catchPhrase":"Multi-layered client-server neural-net","bs":"harness real-time e-markets"}}

file: users.json

[{"id":1,"name":"Leanne Graham","username":"Bret","email":"Sincere@april.biz","address":{"street":"Kulas Light","suite":"Apt. 556","city":"Gwenborough","zipcode":"92998-3874","geo":{"lat":"-37.3159","lng":"81.1496"}},"phone":"1-770-736-8031 x56442","website":"hildegard.org","company":{"name":"Romaguera-Crona","catchPhrase":"Multi-layered client-server neural-net","bs":"harness real-time e-markets"}},{"id":2,"name":"Ervin Howell","username":"Antonette","email":"Shanna@melissa.tv","address":{"street":"Victor Plains","suite":"Suite 879","city":"Wisokyburgh","zipcode":"90566-7771","geo":{"lat":"-43.9509","lng":"-34.4618"}},"phone":"010-692-6593 x09125","website":"anastasia.net","company":{"name":"Deckow-Crist","catchPhrase":"Proactive didactic contingency","bs":"synergize scalable supply-chains"}},{"id":3,"name":"Clementine Bauch","username":"Samantha","email":"Nathan@yesenia.net","address":{"street":"Douglas Extension","suite":"Suite 847","city":"McKenziehaven","zipcode":"59590-4157","geo":{"lat":"-68.6102","lng":"-47.0653"}},"phone":"1-463-123-4447","website":"ramiro.info","company":{"name":"Romaguera-Jacobson","catchPhrase":"Face to face bifurcated interface","bs":"e-enable strategic applications"}},{"id":4,"name":"Patricia Lebsack","username":"Karianne","email":"Julianne.OConner@kory.org","address":{"street":"Hoeger Mall","suite":"Apt. 692","city":"South Elvis","zipcode":"53919-4257","geo":{"lat":"29.4572","lng":"-164.2990"}},"phone":"493-170-9623 x156","website":"kale.biz","company":{"name":"Robel-Corkery","catchPhrase":"Multi-tiered zero tolerance productivity","bs":"transition cutting-edge web services"}},{"id":5,"name":"Chelsey Dietrich","username":"Kamren","email":"Lucio_Hettinger@annie.ca","address":{"street":"Skiles Walks","suite":"Suite 351","city":"Roscoeview","zipcode":"33263","geo":{"lat":"-31.8129","lng":"62.5342"}},"phone":"(254)954-1289","website":"demarco.info","company":{"name":"Keebler LLC","catchPhrase":"User-centric fault-tolerant solution","bs":"revolutionize end-to-end systems"}},{"id":6,"name":"Mrs. Dennis Schulist","username":"Leopoldo_Corkery","email":"Karley_Dach@jasper.info","address":{"street":"Norberto Crossing","suite":"Apt. 950","city":"South Christy","zipcode":"23505-1337","geo":{"lat":"-71.4197","lng":"71.7478"}},"phone":"1-477-935-8478 x6430","website":"ola.org","company":{"name":"Considine-Lockman","catchPhrase":"Synchronised bottom-line interface","bs":"e-enable innovative applications"}},{"id":7,"name":"Kurtis Weissnat","username":"Elwyn.Skiles","email":"Telly.Hoeger@billy.biz","address":{"street":"Rex Trail","suite":"Suite 280","city":"Howemouth","zipcode":"58804-1099","geo":{"lat":"24.8918","lng":"21.8984"}},"phone":"210.067.6132","website":"elvis.io","company":{"name":"Johns Group","catchPhrase":"Configurable multimedia task-force","bs":"generate enterprise e-tailers"}},{"id":8,"name":"Nicholas Runolfsdottir V","username":"Maxime_Nienow","email":"Sherwood@rosamond.me","address":{"street":"Ellsworth Summit","suite":"Suite 729","city":"Aliyaview","zipcode":"45169","geo":{"lat":"-14.3990","lng":"-120.7677"}},"phone":"586.493.6943 x140","website":"jacynthe.com","company":{"name":"Abernathy Group","catchPhrase":"Implemented secondary concept","bs":"e-enable extensible e-tailers"}},{"id":9,"name":"Glenna Reichert","username":"Delphine","email":"Chaim_McDermott@dana.io","address":{"street":"Dayna Park","suite":"Suite 449","city":"Bartholomebury","zipcode":"76495-3109","geo":{"lat":"24.6463","lng":"-168.8889"}},"phone":"(775)976-6794 x41206","website":"conrad.com","company":{"name":"Yost and Sons","catchPhrase":"Switchable contextually-based project","bs":"aggregate real-time technologies"}},{"id":10,"name":"Clementina DuBuque","username":"Moriah.Stanton","email":"Rey.Padberg@karina.biz","address":{"street":"Kattie Turnpike","suite":"Suite 198","city":"Lebsackbury","zipcode":"31428-2261","geo":{"lat":"-38.2386","lng":"57.2232"}},"phone":"024-648-3804","website":"ambrose.net","company":{"name":"Hoeger LLC","catchPhrase":"Centralized empowering task-force","bs":"target end-to-end models"}}]users.json

Basic Filters

Identity: .

This is the simplest basic filter of jq. It takes the input and produces the same value as output. Since jq automatically pretty-prints JSON, this filter can be used to simply format the output in a readable JSON format.

Example: jq '.' user.json

Object Identifier-Index: .foo, .foo.bar

This command searches for objects in the JSON. If the specified object is present, it provides the value at the key “foo”; otherwise, it shows null. Users can provide the JSON path to fetch values from the nested JSON structures.

Examples:

jq '.name' user.json

jq '.profileName' user.json

jq '.address.geo' user.json

Recursive Descent: ..

This command will recursively print the values of all the keys in the JSON.

Example: jq '..' user.json

Array Index: .[<number>]

This command will return the content of the array object based on the given index. Index starts at 0 and supports both positive and negative integers. .[-1] will show the last item in the array object.

Examples:

jq '.[0]' users.json

jq '.[2]' users.json

jq '.[-1]' users.json

Array/String Slice: .[<number>:<number>]

The .[<number>:<number>] syntax is used to return a sub-array of an array or a sub-string of a string. The array returned from .[2:5] will be of length 3, including contents from index 2 (inclusive) to index 5 (exclusive). Indexes are zero-based, and either of the index can be negative (in this case it can count array in the reverse direction from the end of the array). If any of the indexes are missing, it will consider the contents from the beginning or end of the array.

Examples:

jq '.[2:5]' users.json

jq '.[8:]' users.json

jq '.[:-8]' users.json

Array/Object Value Iterator: .[]

This command returns the entire content of an array, by unpacking all the objects from the array and returning them as individual objects. .[1,2,3] will return three separate results at indexes 1, 2, and 3, rather than returning them in a single array.

Example: Let us take a simple json for this example and run the jq with array/object value iterator filter.

# create a simple json object and store in a variable
% sample_data='[{"key1":"value1"},{"key2":"value2"},{"key3":"value3"},{"key4":"value4"}]'

# returns all the entities present in the array
% jq '.[]' <<< $sample_data
{
  "key1": "value1"
}
{
  "key2": "value2"
}
{
  "key3": "value3"
}
{
  "key4": "value4"
}

# returns result in a single array based on given index range
% jq '.[1:4]' <<< $sample_data
[
  {
    "key2": "value2"
  },
  {
    "key3": "value3"
  },
  {
    "key4": "value4"
  }
]

# returns result individually based on given comma-separated indexes
% jq '.[1,2,3]' <<< $sample_data
{
  "key2": "value2"
}
{
  "key3": "value3"
}
{
  "key4": "value4"
}

Comma: ,

If the filters are separated by comma, it checks and fetches the values of the keys, generating the concatenated output based on the expressions from the left to right.

Example: jq '.name, .address, .username' user.json

Pipe: |

This filter will combine filters by providing the output of one of the left into the input of the one of the right. If the left produces multiple results, the right will be executed for each output produced by the left.

Examples:

jq '.[] | .name' users.json .[] filter returns all the user objects of the array and .name will return the name of each user present in the array.

jq '.address | .zipcode' user.json .address will return the address object and .zipcode will return zipcode present inside address object

Parenthesis ()

Parenthesis is used as a grouping operator.

Example: jq '(.[].id * 20)' users.json fetches the id of each user object from array and multiply by 20

Manipulate JSON with filters

updating the content of users.json file and reducing the size of user object

[{"id":1,"email":"george.bluth@reqres.in","first_name":"George","last_name":"Bluth","avatar":"https://reqres.in/img/faces/1-image.jpg"},{"id":2,"email":"janet.weaver@reqres.in","first_name":"Janet","last_name":"Weaver","avatar":"https://reqres.in/img/faces/2-image.jpg"},{"id":3,"email":"emma.wong@reqres.in","first_name":"Emma","last_name":"Wong","avatar":"https://reqres.in/img/faces/3-image.jpg"},{"id":4,"email":"eve.holt@reqres.in","first_name":"Eve","last_name":"Holt","avatar":"https://reqres.in/img/faces/4-image.jpg"},{"id":5,"email":"charles.morris@reqres.in","first_name":"Charles","last_name":"Morris","avatar":"https://reqres.in/img/faces/5-image.jpg"}]

Now that we are familiar with the basic filters, let's experiment with JSON content and manipulate it using these filters.

• add key “character” to each user object in users.json jq '.[] | .character = "cartoon"' users.json

• rename the id and first_name fields in the JSON jq '.[] | {index: .id, firstName: .first_name}' users.json

• print the full name of each user by combining the first_name and last_name fields jq '.[] | "Full Name of user is (.first_name) (.last_name)"' users.json

Built-in functions

jq supports multiple built-in functions. However, we will cover only a few of them.

keys

keys return the key of each object in the JSON jq '.[] | keys' users.json

values

this function will return the value of each key in the object

length

returns the length of the given array object or the length of the value of each key in object.

jq '.|length' users.json

jq '.[]|length' user.json

map(f)

for any filter f, map(f) will apply for each value of f

jq '.| map(.id*=5)' users.json

jq '.| map(.first_name += "My First Name )")' users.json

add “My First Name “ prefix to first_name field

select(boolean_expression)

select(f) will return the value based on the condition of the filter f. If f is true return value else return nothing.

jq '.[]| select(.id >= 2)' users.json

There are many other built-in functions that are quite helpful and will be covered in upcoming articles.

Reference

https://jqlang.github.io/jq/

Conclusion

In conclusion, jq is an invaluable tool for anyone working with JSON data on the command line. Its ease of installation, flexibility, and powerful filtering capabilities make it a preferred choice for developers, testers, and data analysts. By mastering jq's basic filters and built-in functions, users can efficiently parse, manipulate, and transform JSON data to suit their needs. Whether you're dealing with simple JSON files or complex nested structures, jq provides the functionality needed to streamline your workflow and enhance productivity. As you continue to explore jq, you'll discover even more advanced features and techniques that can further optimize your data processing tasks.