How to extract a value from a JSON file containing an encoded JSON object

awsjqjson

The command aws s3api get-bucket-policy --bucket bucketname
outputs:

{
    "Policy": "{\"Version\":\"2012-10-17\",\"Id\":\"S3SecureTransportPolicy\",\"Statement\":[{\"Sid\":\"ForceSSLOnlyAccess\",\"Effect\":\"Deny\",\"Principal\":{\"AWS\":\"*\"},\"Action\":\"s3:*\",\"Resource\":\"arn:aws:s3:::amn/*\",\"Condition\":{\"Bool\":{\"aws:SecureTransport\":\"false\"}}},{\"Sid\":\"AWSCloudTrailAclCheck20150319\",\"Effect\":\"Allow\",\"Principal\":{\"Service\":\"cloudtrail.amazonaws.com\"},\"Action\":\"s3:GetBucketAcl\",\"Resource\":\"arn:aws:s3:::amn\"},{\"Sid\":\"AWSCloudTrailWrite20150319\",\"Effect\":\"Allow\",\"Principal\":{\"Service\":\"cloudtrail.amazonaws.com\"},\"Action\":\"s3:PutObject\",\"Resource\":\"arn:aws:s3:::amn/AWSLogs/405042254276/*\",\"Condition\":{\"StringEquals\":{\"s3:x-amz-acl\":\"bucket-owner-full-control\"}}}]}"
}

I need to grep (i.e., extract) the value associated with the key aws:SecureTransport from this JSON (it would be false, in this case) and the value associated with the key Effect.

I've tried

aws s3api get-bucket-policy --bucket amn |
    grep -Po '"Bool": *\K"[^"]*"'

and

aws s3api get-bucket-policy --bucket amn |
    sed 's/.*\(aws:SecureTransport\)[^:]*:"\([0-9]*\)"'

How can I do this?

Best Answer

The JSON document that you get from your command seems to contain another encoded JSON document. It's from this encoded document you appear to want to get the data.

To get at the internal document, we may use jq:

aws ... |
jq -r '.Policy'

To get the value of the Effect key from the bit that contains that aws:SecureTransport key from this, we need to parse the document again:

aws ... |
jq -r '.Policy' |
jq -r '.Statement[] | select(.Condition.Bool."aws:SecureTransport").Effect'

The last jq call goes through all the elements of the Statement array, looking for one that has a key called .Condition.Bool."aws:SecureTransport". It then gets the value of the Effect key associated with that Statement element.

Running this on your data outputs the value Deny.

If you want the value of that .Condition.Bool."aws:SecureTransport" key (false in your document), use .Condition.Bool."aws:SecureTransport" in place of .Effect above.

Alternatively, use the fromjson instruction in jq instead of a second jq invocation:

aws ... |
jq -r '.Policy | fromjson | .Statement[] | select(.Condition.Bool."aws:SecureTransport").Effect'

Here, fromjson decodes the encoded JSON document and passes it to the later stages of processing.

Just for reference, the internal encoded JSON document looks like this (aws ... | jq -r '.Policy | fromjson'):

{
  "Version": "2012-10-17",
  "Id": "S3SecureTransportPolicy",
  "Statement": [
    {
      "Sid": "ForceSSLOnlyAccess",
      "Effect": "Deny",
      "Principal": {
        "AWS": "*"
      },
      "Action": "s3:*",
      "Resource": "arn:aws:s3:::amn/*",
      "Condition": {
        "Bool": {
          "aws:SecureTransport": "false"
        }
      }
    },
    {
      "Sid": "AWSCloudTrailAclCheck20150319",
      "Effect": "Allow",
      "Principal": {
        "Service": "cloudtrail.amazonaws.com"
      },
      "Action": "s3:GetBucketAcl",
      "Resource": "arn:aws:s3:::amn"
    },
    {
      "Sid": "AWSCloudTrailWrite20150319",
      "Effect": "Allow",
      "Principal": {
        "Service": "cloudtrail.amazonaws.com"
      },
      "Action": "s3:PutObject",
      "Resource": "arn:aws:s3:::amn/AWSLogs/405042254276/*",
      "Condition": {
        "StringEquals": {
          "s3:x-amz-acl": "bucket-owner-full-control"
        }
      }
    }
  ]
}
Related Question