Step 10 · Ubuntu · Practice

Doing more with jq

In step 2 you used jq to check whether a file was valid JSON. That is the smallest thing it does. This page is for when you want to pull information out of JSON — reading fields, filtering lists, counting and reshaping.

Nothing to install

jq came with step 2. This page is practice only — there is no script to run. Work through it whenever you are ready.

The dot means “the whole thing”

Every jq command is a small expression describing what you want. The simplest expression is ., which means “everything” — that is why jq . file.json prints the file back unchanged.

Start from the file you made in step 2:

{
  "name": "Lakshmi Prasad",
  "roll_number": "24CS042",
  "image_url": "https://example.com/photos/lakshmi.jpg",
  "sankalpam": "I want to be an expert software builder who solve problems with software solutions plus services"
}

Add a key name after the dot to pull out just that value:

$ jq .name student.json
"Lakshmi Prasad"

$ jq .roll_number student.json
"24CS042"

$ jq .sankalpam student.json
"I want to be an expert software builder who solve problems with software solutions plus services"

Losing the quotation marks

$ jq -r .name student.json
Lakshmi Prasad

What -r does, and why you want it

Without it, jq prints text values with their JSON quotes: "Lakshmi Prasad". -r gives raw output with no quotes, which is what you need when feeding the result into another command.

Several fields at once

# two values, one per line
$ jq -r '.name, .roll_number' student.json

# build a smaller object from the ones you want
$ jq '{name, roll_number}' student.json

# rename a key on the way out
$ jq '{student: .name, roll: .roll_number}' student.json

# list the keys the file contains
$ jq 'keys' student.json

Working with a list

JSON gets more interesting when a file holds many records. Create this one to practise on — it works offline, with no network needed:

$ cat > readings.json <<'JSON'
{
  "site": "north",
  "readings": [
    {"id": 1, "value": 12, "ok": true},
    {"id": 2, "value": 7,  "ok": true},
    {"id": 3, "value": 21, "ok": false},
    {"id": 4, "value": 9,  "ok": true}
  ]
}
JSON

Then work through these in order. Each one builds on the last.

#	Task	Command	Answer
1	Print the site name	`jq -r .site readings.json`	`north`
2	Every value, one per line	`jq '.readings[].value' readings.json`	`12 7 21 9`
3	Only the readings marked ok	`jq '.readings[] \| select(.ok)' readings.json`	3 records
4	How many are not ok?	`jq '[.readings[] \| select(.ok == false)] \| length' readings.json`	`1`
5	Add the values up	`jq '[.readings[].value] \| add' readings.json`	`49`
6	The largest value	`jq '[.readings[].value] \| max' readings.json`	`21`
7	Reshape into a simpler list	`jq '[.readings[] \| {id, value}]' readings.json`	`[{"id":1,"value":12}, …]`

The two pieces of punctuation that do the work

[] after a list means “each item in turn”. So .readings[].value reads the value of every reading.
| passes the result of the left side into the right side — the same idea as a pipe in the shell.

Why tasks 4 to 6 wrap everything in [ ]

.readings[].value produces four separate results, not one list of four. length, add and max need a single list to work on, so the square brackets gather the separate results back into one. Leaving them out is the mistake almost everyone makes first.

Reading a real API

Most web services answer in JSON, so curl and jq work together naturally:

# raw JSON: technically readable, practically not
$ curl -s https://api.github.com/repos/aikaryashala/system_setup

# the same thing, indented and coloured
$ curl -s https://api.github.com/repos/aikaryashala/system_setup | jq

# pull out one field
$ curl -s https://api.github.com/repos/aikaryashala/system_setup | jq -r .full_name
aikaryashala/system_setup

# several fields, as a new object
$ curl -s https://api.github.com/repos/aikaryashala/system_setup \
    | jq '{name: .name, stars: .stargazers_count, language: .language}'

# walk into a list: the name of every public repo
$ curl -s https://api.github.com/users/aikaryashala/repos | jq -r '.[].name'

# filter a list, then count what is left
$ curl -s https://api.github.com/users/aikaryashala/repos \
    | jq '[.[] | select(.fork == false)] | length'

Cheat sheet

Expression	Means
`.`	The whole document — use it to check a file is valid
`.name`	The value of the `name` key
`.a.b`	Go one level deeper
`.items[]`	Each item of a list in turn
`.items[0]`	Just the first item
`keys`	The list of keys
`length`	How many items, or how many characters
`add`	Add a list of numbers together
`min` / `max`	Smallest / largest in a list
`select(.ok)`	Keep only the items where that is true
`{a, b}`	Build a new object from those keys
`sort_by(.value)`	Order a list by one field
`-r`	Print text without quotation marks
`-c`	Print each result on one line