Zsh – Assign Data Extracted from JSON Document to Shell Parameter

jqjsonzsh

During everyday shell sessions, I very often find myself needing to assign data from a JSON document (extracted via some jq filter) to a Zsh shell parameter: JSON scalars to Zsh scalars, JSON arrays to Zsh arrays, and JSON objects to Zsh associative arrays. The problem—and what previously asked questions don’t seem to tackle—is that the data often contains newlines (and even NUL bytes), making this a rather nontrivial task.

Here is what I have come up with so far:

function assign-from-json {
    local -A opts && zparseopts -A opts -D -F -M -- a A && typeset -r opts
    if [[ $# -ne 3 || ( -v opts[-a] && -v opts[-A] ) ]] ; then
        >&2 printf 'Usage: %s [-a|-A] NAME FILTER JSON\n' $0
        return 2
    fi
    if [[ -v opts[-a] ]] ; then
        local -a lengths && { lengths=( "${(@f)$( jq -r "$2 | .[] | tostring | length" <<< $3 )}" ) || return $? } && typeset -r lengths
        local data && { data="$( jq -j "$2 | .[] | tostring" <<< $3 )" || return $? } && typeset -r data
        local elem
        local -a elems
        for length in "${lengths[@]}" ; do
            read -u 0 -k $length elem
            elems+=$elem
        done <<< $data
        eval "${(q)1}"='( "${elems[@]}" )'
    elif [[ -v opts[-A] ]] ; then
        local transformed_json && { transformed_json="$( jq "$2 | to_entries | map(.key, .value)" <<< $3 )" || return $? } && typeset -r transformed_json
        assign-from-json -a $1 "." $transformed_json
    else
        eval "${(q)1}"="${(q)$( jq -r $2 <<< $3 )}"
    fi
}

In most cases it works quite well:

% json='
{
    "scalar": "Hello, world",
    "array": [1, 2, 3],
    "scary_scalar": "\nNewlines\u0000NUL bytes\ttabs",
    "scary_array": [
        "A\nvery\u0000scary\nvalue",
        "A less scary value",
        "eh"
    ]
}
'
% assign-from-json scalar '.scalar' $json && printf '%q\n' $scalar
Hello,\ world
% typeset -a array && assign-from-json -a array '.array' $json && printf '%q\n' "${array[@]}"
1
2
3
% assign-from-json scary_scalar '.scary_scalar' $json && printf '%q\n' $scary_scalar
$'\n'Newlines$'\0'NUL\ bytes$'\t'tabs
% typeset -a scary_array && assign-from-json -a scary_array '.scary_array' $json && printf '%q\n' "${scary_array[@]}"
A$'\n'very$'\0'scary$'\n'value
A\ less\ scary\ value
eh
% typeset -A assoc && assign-from-json -A assoc '.' $json && printf '%q -> %q\n' "${(@kv)assoc}"
array -> \[1,2,3\]
scary_array -> \[\"A\\nvery\\u0000scary\\nvalue\",\"A\ less\ scary\ value\",\"eh\"\]
scary_scalar -> $'\n'Newlines$'\0'NUL\ bytes$'\t'tabs
scalar -> Hello,\ world

Unfortunately it seems to struggle with trailing newlines:

% assign-from-json bad_scalar '.' '"foo\n"' && printf '%q\n$ $bad_scalar
foo
# expected: foo$'\n'
  1. I assume the problem with trailing newlines is due to command substitution removing them. Do you see an easy way to fix it?
  2. One can do assign-from-json -A assoc ... even if assoc is not typeset as an associative array. How can I prevent that from being possible?
  3. Do you see any other problems with the code?

Best Answer

A common workaround for keeping trailing newlines in a command substitution is to append a value, so that the newlines are no longer trailing. The dummy value is then removed from the variable:

v1=$'line1\nline2\n\n\n'
v2=$(print $v1)
v3=$(printf $v1; print X);v3=${v3%X}
typeset -p v1 v2 v3

Output:

typeset v1=$'line1\nline2\n\n\n'
typeset v2=$'line1\nline2'
typeset v3=$'line1\nline2\n\n\n'

This does complicate handling return codes, so you might need something like this:

local data \
  && {data="$( jq -j "$2 | .[] | tostring" <<< $3;
    ret=$?;
    print X;
    return $ret)" \
    || return $? } \
  && data=${data:%X} \
  && typeset -r data

Some other options for retaining trailing newlines are listed in this answer. Unfortunately, I don't think zsh has a 'command substitution flag' for situations like this (yet).

The t parameter expansion flag can be used to test the type of a variable and determine if a name references an associative array or something else:

function vtype {
  tt=${(Pt)1}
  if [[ $tt == association* ]]; then
      print "$1: YES [$tt]"
  else
      print "$1: no  [$tt]"
  fi
}
typeset -A a1; vtype a1
typeset -a a2; vtype a2
integer i1; vtype i1
typeset -AlxRr a3; vtype a3

The code above also uses the P expansion flag to interpret the positional parameter value in $1 as yet another parameter name. The test uses a wildcard (...*) because the type identifier from t may have multiple components. The output:

a1: YES [association]
a2: no  [array]
i1: no  [integer]
a3: YES [association-right_blanks-lower-readonly-export]

Good luck getting zsh to do what you need. I often find that stretching a tool slightly beyond its comfortable limits works better than completely rebuilding with a separate, less familiar tool. The hard part is defining 'slightly beyond'.

Related Question