drawing of cat

Adventures in Learning Full Stack Web Development

Pipe it to jq

2019.09.29

What is jq?

I use jq when I need a quick way to gain insights on JSON in the command line. According to the jq manual, “a jq program is a “filter”: it takes an input, and produces an output.” This post will provide examples of how jq can help you answer questions about JSON. I recommend installing jq and trying out the examples below in your terminal.

I’m going to use the Google Books API to query a list of quilting books. I’ll be using curl to get the JSON data. Let’s see what JSON data we are working with by curling the API endpoint.

🚨 The data returned in the examples below may change over time as the API response changes.

curl 'https://www.googleapis.com/books/v1/volumes?q=quilting'

We have 3 objects. The last object holds an array of item objects. An item object represents a quilting book.

{
  "kind": "books#volumes",
  "totalItems": 1916,
  //array of quilting book objects
  "items": []
{

Piping to jq

Different jq commands are “piped” to JSON output. The simplest command (| jq) pretty prints the JSON output, which makes the JSON more readable. This command saves you time because you no longer need to copy the output and format it in a text editor just to make it more readable.

curl 'https://www.googleapis.com/books/v1/volumes?q=quilting' | jq

If you are following along, you can see the JSON is enhanced with color.

Answering simple questions

  • How many JSON objects are returned?

You can pipe to the length command to find the answer:

curl 'https://www.googleapis.com/books/v1/volumes?q=quilting' | jq 'length'

3

  • What are the keys?
curl 'https://www.googleapis.com/books/v1/volumes?q=quilting' | jq 'keys'

[
  "items",
  "kind",
  "totalItems"
]
  • How can we look at only the items?
curl 'https://www.googleapis.com/books/v1/volumes?q=quilting' | jq '.items'

This response is too long to share here, but it returns only the contents of the “items” key, which is an array of book/item objects.

  • How can we look at only the first item in the items array?
curl 'https://www.googleapis.com/books/v1/volumes?q=quilting' | jq '.items[0]'

{
  "kind": "books#volume",
  "id": "TBxRAgAAQBAJ",
  "etag": "ChSl05kO0Tw",
  "selfLink": "https://www.googleapis.com/books/v1/volumes/TBxRAgAAQBAJ",
  "volumeInfo": {
    "title": "Quilting with a Modern Slant",
    "subtitle": "People, Patterns, and Techniques Inspiring the Modern Quilt Community",
    "authors": [
      "Rachel May",
      "Rachel Suzanne May"
    ],
    "publisher": "Storey Publishing",
    "publishedDate": "2014-01-01",
    "description": "Profiles more than seventy modern quilters, offering step-by-step instructions on their techniques and quilting projects.",
  },
  "saleInfo": {
    "country": "US",
    "saleability": "NOT_FOR_SALE",
    "isEbook": false
  },
  "accessInfo": {
    "country": "US",
    "viewability": "PARTIAL",
    "webReaderLink": "http://play.google.com/books/reader?id=TBxRAgAAQBAJ&hl=&printsec=frontcover&source=gbs_api",
  },
  "searchInfo": {
    "textSnippet": "Tour the varied world of modern quilting through the stories and works of more than 70 of its most exciting artists."
  }
}

If you are following along, some of the data has been omitted to make this post easier to read, but all the data we need for this post has been included.

Chaining jq filters to answer more complex questions

  • How many total items are there?
curl 'https://www.googleapis.com/books/v1/volumes?q=quilting' | jq '.items | length'

10

However, if you change it slightly, you’ll be answering a different question:

  • How many total objects are in the each objects in the items array?
curl 'https://www.googleapis.com/books/v1/volumes?q=quilting' | jq '.items[] | length'

8
8
8
8
8
8
8
8
8
8

Notice the array/object value iterator (.[]) was appended to the '.items' argument. Running the array/object value iterator with the input of the items array will produce each item as a separate result, as opposed to a single array. Each item object in the array includes 8 objects, which is why 8 is returned for each item in the items array. One way that we can confirm the number of objects is by getting the keys for the first object.

  • What are the keys of the first object?
curl 'https://www.googleapis.com/books/v1/volumes?q=quilting' | jq '.items[0] | keys'

[
  "accessInfo",
  "etag",
  "id",
  "kind",
  "saleInfo",
  "searchInfo",
  "selfLink",
  "volumeInfo"
]

You can see that there are 8 keys. If there were too many keys to quickly count, you could pipe to the length filter to get the total number of keys.

  • How many keys are in the first object in the items array?
curl 'https://www.googleapis.com/books/v1/volumes?q=quilting' | jq '.items[0] | keys | length'

8

  • What is the selfLink for each book item in the items array?
curl 'https://www.googleapis.com/books/v1/volumes?q=quilting' | jq '.items[].selfLink'

"https://www.googleapis.com/books/v1/volumes/TBxRAgAAQBAJ"
"https://www.googleapis.com/books/v1/volumes/Ym4usUyOmNEC"
"https://www.googleapis.com/books/v1/volumes/PyPWxkmifJQC"
"https://www.googleapis.com/books/v1/volumes/fUgj6cpxXJYC"
"https://www.googleapis.com/books/v1/volumes/kGhFfARr_o4C"
"https://www.googleapis.com/books/v1/volumes/K-Zhvd-kMy8C"
"https://www.googleapis.com/books/v1/volumes/SVJiyOfIpoMC"
"https://www.googleapis.com/books/v1/volumes/UgFpAAAACAAJ"
"https://www.googleapis.com/books/v1/volumes/vXMz6hFSlkwC"
"https://www.googleapis.com/books/v1/volumes/VJCbqZoX-HwC"
  • What is the title of each book item?
curl 'https://www.googleapis.com/books/v1/volumes?q=quilting' | jq '.items[].volumeInfo.title'

"Quilting with a Modern Slant"
"Free-Motion Quilting with Angela Walters"
"Pat Sloan's I Can't Believe I'm Quilting"
"5,500 Quilt Block Designs"
"Machine Quilting Solutions"
"Listen to Your Quilt"
"Machine Quilting Made Easy"
"Quilting Makes the Quilt"
"Hand & Machine Quilting Tips & Tricks Tool"
"Show Me How To Plan My Quilting"
  • Which books are ebooks?
curl 'https://www.googleapis.com/books/v1/volumes?q=quilting' | jq '.items[].saleInfo.isEbook'

false
true
false
false
true
false
false
false
true

Hmm, that’s not very useful. It would be nice to know the specific book titles, and this just returns a boolean without the book object. If we had a larger number of objects, we wouldn’t be able to count and see that the number of ebooks. If we use the select filter we can filter the result to return only books that are ebooks and then count those books by getting the length of the array that’s returned. Also note, the results of the select is wrapped in square brackets to ensure we can get the length of the final array.

curl 'https://www.googleapis.com/books/v1/volumes?q=quilting' | jq '[.items[] | select(.saleInfo.isEbook == true)] | 'length''

4

Final notes

If you ever want to have more readable JSON in your command line or you want to filter your data to learn more about it, pipe it to jq. One key to jq is understanding how to chain filters together to get the specific answers you want. The best way to learn this is to practice. There is a great tutorial in the jq documentation and here is a cheat sheet that you might find useful.