Curl JSON encoded in UTF-8

character encodingcygwin;jsonunicode

Important note: I'm using Cygwin

I retrieve JSON from a file that I alter before sending it to a server using curl. By now everything is ok except one thing, when the server receive the JSON, every special characters (with accents, etc) aren't well encoded. I suppose this was due to the fact that my JSON wasn't encoded in UTF-8 before being sent but I don't manage to convert it.

Here is the code:

sed -i 's/\r//g' $file
fileContent=`cat $file`

result=$(jq -c -j ".docs[$docIndex] + { \"_rev\": \"$rev\" }"<<<"$fileContent")            result="{\"docs\":[$result]}"

result=$result | sed 's/\r//g'
result=$result | iconv -t "UTF-8"
s=$(curl -H "Content-Type: application/json; charset=UTF-8" -H "Cache-Control: no-cache" -d "$result" $2/$3/_bulk_docs --silent)

My bash script and my JSON files are both encoded in UTF-8.
My LANG variable seems to be UTF-8. I checked with this: [[ $LANG =~ UTF-8$ ]] && echo "Uses UTF-8 encoding.."

Any idea?

Update

Here is the full script:

#!/bin/bash

# $1 = directory containing JSON files
# $2 = server url (+ port)
# $3 = database name

#Loop on every file of the given directory
for file in "$1"/*; do

    #Try to insert every document of a specific JSON file and retrieve their status (200 = OK; 409 = conflict)
    allStatus=$(curl -H "Content-Type: application/json" -H "Cache-Control: no-cache" --data-binary "@$file" $2/$3/_bulk_docs --silent | jq '.[] |.status' | tr -d '\r')
    docIndex=0

    #Loop on every status (one status = one document)
    while IFS=' ' read -ra statusArray; do
      for status in "${statusArray[@]}"; do

        #Remove unwanted windows characters of the file
        sed -i 's/\r//g' $file
        fileContent=`cat $file`

        #Retrieve the id of the current document
        id=`jq -r -j ".docs[$docIndex]._id"<<<"$fileContent" | tr -d '\r'`

        if [ "$status" = "409" ]
        then 
            #Retrieve the revision of the current document and add it to the document 
            rev=$(curl -X GET --header 'Accept: application/json' $2/$3/$id?revs=true --silent | jq -r -j '._rev' | tr -d '\r')
            result=$(jq -c -j ".docs[$docIndex] + { \"_rev\": \"$rev\" }"<<<"$fileContent")

            #Wrap the document inside {"docs":[]}
            result="{\"docs\":[$result]}"

            #Remove unwanted windows characters before sending the document (again)
            result=$result | sed 's/\r//g'
            result=$result | iconv -t "UTF-8"
            s=$(curl -H "Content-Type: application/json; charset=UTF-8" -H "Cache-Control: no-cache" -d "$result" $2/$3/_bulk_docs --silent)

            #Echo result
            echo $s
        else
          #Status should be 200.
          echo 'status: '$status ' - ' $id
        fi

        docIndex=$((docIndex+1))

      done
    done <<< "$allStatus"
done

This script aims to upsert documents inside a NoSQL database. If first try to insert and if if fails it retrieves a property (the revision) of the document, append to it and then retry once.

I know this script could be improved but this is my first bash script and it won't be used in production (only for testing, etc.).

Best Answer

You can use this in Bash on Cygwin, as long as you at least have Python 2.7 installed:

utf8_result="$(echo "$result" | python -c "import sys;[sys.stdout.write((i).decode('utf-8')) for i in sys.stdin]")"

Related Solutions

How to set VIM’s default encoding to UTF-8

When Vim reads an existing file, it tries to detect the file encoding. When writing out the file, Vim uses the file encoding that it detected (except when you tell it differently). So a file detected as UTF-8 is written as UTF-8, a file detected as Latin-1 is written as Latin-1, and so on.

By default, the detection process is crude. Every file that you open with Vim will be assumed to be Latin-1, unless it detects a Unicode byte-order mark at the top. A UTF-8 file without a byte-order mark will be hard to edit because any multibyte characters will be shown in the buffer as character sequences instead of single characters.

Worse, Vim, by default, uses Latin-1 to represent the text in the buffer. So a UTF-8 file with a byte-order mark will be corrupted by down-conversion to Latin-1.

The solution is to configure Vim to use UTF-8 internally. This is, in fact, recommended in the Vim documentation, and the only reason it is not configured that way out of the box is to avoid creating enormous confusion among users who expect Vim to operate basically as a Latin-1 editor.

In your .vimrc, add set encoding=utf-8 and restart Vim.

Or instead, set the LANG environment variable to indicate that UTF-8 is your preferred character encoding. This will affect not just Vim but any software which relies on LANG to determine how it should represent text. For example, to indicate that text should appear in English (en), as spoken in the United States (US), encoded as UTF-8 (utf-8), set LANG=en_US.utf-8.

Now Vim will use UTF-8 to represent the text in the buffer. Plus, it will also make a more determined effort to detect the UTF-8 encoding in a file. Besides looking for a byte-order mark, it will also check for UTF-8 without a byte-order mark before falling back to Latin-1. So it will no longer corrupt a file coded in UTF-8, and it should properly display the UTF-8 characters during the editing session.

For more information on how Vim detects the file encoding, see the fileencodings option in the Vim documentation.

For more information on setting the encoding that Vim uses internally, see the encoding option.

If you need to override the encoding used when writing a file back to disk, see the fileencoding option.

Bash – Sanitize JSON for sending with Curl

Using the jo utility, the sanitized JSON document could be constructed using

data=$( jo Attribute="$USERINPUT" )

You would then use -d "$data" with curl to pass this to the end point.

Old answer using jq instead of jo:

Using jq:

USERINPUT=$'a e""R<*&\04\n\thello!\''

This string has a couple of double quotes, an EOT character, a newline, a tab and a single quote, along with some ordinary text.

data="$( jq --null-input --compact-output --arg str "$USERINPUT" '{"Attribute": $str}' )"

This builds a JSON object containing the user data as the value for the lone Attribute field.

The same thing using short options:

data="$( jq -nc --arg str "$USERINPUT" '{"Attribute": $str}' )"

From this we get

{"Attribute":"ae\"\"R<*&\u0004\n\thello!'"}

as the value in $data.

This can now be used in your call to curl:

RESP="$( curl --connect-timeout "10" -s \
           -H "Content-Type: application/json" \
           -X POST -d "$data" \
           "$ENDPOINT" | jq -r '.key' )"

Best Answer

Related Solutions

How to set VIM’s default encoding to UTF-8

Bash – Sanitize JSON for sending with Curl

Related Question