Shell – How to Merge Multiple JSON Objects into One Array

jqjsonlinux

I have a requirement to read some fields of json data from a file to compose a new json list. The file is generated by reading data from multiple servers via pssh

# pssh_result.txt
[1] 14:44:51 [SUCCESS] 1.1.1.1
{"field1":"value11","field2":"value12","field3":"value13",...}
[2] 14:44:51 [SUCCESS] 1.1.1.2
{"field1":"value21","field2":"value22","field3":"value23",...}
...
...

I can only extract fields from it at the moment, but I don't know how to combine them into a finished json array

#!/bin/sh
input="pssh_result.txt"
while IFS= read -r line; do
    # echo "$line"
    if jq -e . >/dev/null 2>&1 <<<"$line"; then
        read field1 field3 <<<$(echo $(echo $line |
            jq -r '.field1, .field3'))
        example_item=$(jq --null-input \
            --arg field1 "$field1" \
            --arg field3 "$field3" \
            '{"alias1": $field1, "alias2": $field3}')

        echo $example_item
    fi
done <"$input"

the output are as follow

sh test.sh 
{ "alias1": "value11", "alias2": "value13" }
{ "alias1": "value21", "alias2": "value23" }

But I want to merge it into a large json array for later sending to a third party for processing, and the desired result is as follows

[
    {
        "alias1": "value11",
        "alias2": "value13"
    },
    {
        "alias1": "value21",
        "alias2": "value23"
    }
]

I also thought about putting it into a array and then passing it through something like join(arr,","), but it didn't work as expected.

How do I get the results I want? I really appreciate any help with this.

Best Answer

Assuming that the input looks like

[1] 14:44:51 [SUCCESS] 1.1.1.1
{"field1":"value11","field2":"value12","field3":"value13"}
[2] 14:44:51 [SUCCESS] 1.1.1.2
{"field1":"value21","field2":"value22","field3":"value23"}

(Note that I have fixed the broken JSON on the lines that have JSON objects.)

... then you can extract the lines that you want with grep:

grep '^{' file

or

grep -v -F '[SUCCESS]' file

or similar grep command.

You may then parse the stream of JSON objects using jq -s, which will automatically read them into an array. This allows you do the processing of the data like so:

grep '^{' file | jq -s 'map({ alias1: .field1, alias2: .field3 })'

With the input shown above in file, this would generate

[
  {
    "alias1": "value11",
    "alias2": "value13"
  },
  {
    "alias1": "value21",
    "alias2": "value23"
  }
]

If you are uninterested in changing your already existing initial parsing of the input and want to put your output into an array, then it's just a matter of passing it through jq -s .:

$ cat file
{ "alias1": "value11", "alias2": "value13" }
{ "alias1": "value21", "alias2": "value23" }

$ jq -s . file
[
  {
    "alias1": "value11",
    "alias2": "value13"
  },
  {
    "alias1": "value21",
    "alias2": "value23"
  }
]

A hack to read directly from the original file with jq is to read the data as raw strings, try to convert them into valid JSON, and then throw away any errors:

$ cat file
[1] 14:44:51 [SUCCESS] 1.1.1.1
{"field1":"value11","field2":"value12","field3":"value13"}
[2] 14:44:51 [SUCCESS] 1.1.1.2
{"field1":"value21","field2":"value22","field3":"value23"}

$ jq -R 'fromjson | { alias1: .field1, alias2: .field3 }' file 2>/dev/null
{
  "alias1": "value11",
  "alias2": "value13"
}
{
  "alias1": "value21",
  "alias2": "value23"
}

This could then be gathered in an array:

$ jq -R 'fromjson | { alias1: .field1, alias2: .field3 }' file 2>/dev/null | jq -s .
[
  {
    "alias1": "value11",
    "alias2": "value13"
  },
  {
    "alias1": "value21",
    "alias2": "value23"
  }
]

Note that we need the two separate invocations of jq here as the first jq invocation would terminate prematurely if it tried to convert the input into JSON with fromjson into an array (or any other structure other than individual objects).

Related Question