JQ – How to Merge Arrays from Multiple JSON Files

Assuming that the input looks like

[1] 14:44:51 [SUCCESS] 1.1.1.1
{"field1":"value11","field2":"value12","field3":"value13"}
[2] 14:44:51 [SUCCESS] 1.1.1.2
{"field1":"value21","field2":"value22","field3":"value23"}

(Note that I have fixed the broken JSON on the lines that have JSON objects.)

... then you can extract the lines that you want with grep:

grep '^{' file

or

grep -v -F '[SUCCESS]' file

or similar grep command.

You may then parse the stream of JSON objects using jq -s, which will automatically read them into an array. This allows you do the processing of the data like so:

grep '^{' file | jq -s 'map({ alias1: .field1, alias2: .field3 })'

With the input shown above in file, this would generate

[
  {
    "alias1": "value11",
    "alias2": "value13"
  },
  {
    "alias1": "value21",
    "alias2": "value23"
  }
]

If you are uninterested in changing your already existing initial parsing of the input and want to put your output into an array, then it's just a matter of passing it through jq -s .:

$ cat file
{ "alias1": "value11", "alias2": "value13" }
{ "alias1": "value21", "alias2": "value23" }

$ jq -s . file
[
  {
    "alias1": "value11",
    "alias2": "value13"
  },
  {
    "alias1": "value21",
    "alias2": "value23"
  }
]

A hack to read directly from the original file with jq is to read the data as raw strings, try to convert them into valid JSON, and then throw away any errors:

$ cat file
[1] 14:44:51 [SUCCESS] 1.1.1.1
{"field1":"value11","field2":"value12","field3":"value13"}
[2] 14:44:51 [SUCCESS] 1.1.1.2
{"field1":"value21","field2":"value22","field3":"value23"}

$ jq -R 'fromjson | { alias1: .field1, alias2: .field3 }' file 2>/dev/null
{
  "alias1": "value11",
  "alias2": "value13"
}
{
  "alias1": "value21",
  "alias2": "value23"
}

This could then be gathered in an array:

$ jq -R 'fromjson | { alias1: .field1, alias2: .field3 }' file 2>/dev/null | jq -s .
[
  {
    "alias1": "value11",
    "alias2": "value13"
  },
  {
    "alias1": "value21",
    "alias2": "value23"
  }
]

Note that we need the two separate invocations of jq here as the first jq invocation would terminate prematurely if it tried to convert the input into JSON with fromjson into an array (or any other structure other than individual objects).

The following uses the relational JOIN() function in jq to join the two result arrays on the elements that are equal with respect to the lower-case variant of the name key (servers) and the cmdb.name key (IPs). It also uses INDEX() to build an index of the IP file's result array. The JOIN() function gives us arrays (pairs in the example) of matching objects that we merge using the add function. After joining and merging, we are left with objects containing both the name and the cmdb.name keys, so we delete the latter in each object.

jq '.result = [JOIN(INDEX(input.result[]; ."cmdb.name"|ascii_downcase); .result[]; .name|ascii_downcase) | add | del(."cmdb.name")]' servers.json ips.json

The jq expression, nicely formatted:

.result =
  [
    JOIN(
      # index on the second file's .cmdb.name key in each result object
      INDEX(
        input.result[];
        ."cmdb.name" | ascii_downcase
      );
      .result[];             # join on the first file's result objects
      .name | ascii_downcase # match using the .name key
    )
    | add               # merge the matched objects
    | del(."cmdb.name") # delete that key we don't want
  ]

Result:

{
  "result": [
    {
      "os": "Microsoft Windows Server 2019 Standard",
      "name": "SERVER1",
      "ip_address": "10.0.0.10",
      "interface": "Intel Wireless-AC 9560 160MHz"
    },
    {
      "os": "Microsoft Windows Server 2019 Standard",
      "name": "SERVER2",
      "ip_address": "10.0.0.10",
      "interface": "Wi-Fi"
    },
    {
      "os": "Microsoft Windows Server 2019 Standard",
      "name": "server3",
      "ip_address": ""
    },
    {
      "os": "Microsoft Windows Server 2016 Standard",
      "name": "server4",
      "ip_address": "10.0.0.10",
      "interface": "Intel Dual Band Wireless-AC 8265"
    }
  ]
}