Shell – Unable to pass a bash variable as a python argument in bash

amazon s3pythonshell-script

For some reason I cannot pass the bash variable $FOLDER as a python argument on the following code. The python script downloads some files from amazon s3.

Bash script:

#!/bin/bash
FOLDER=$(./aws get $BUCKET"/location.txt")
python /data/pythonscript.py $FOLDER
#

The output of the $FOLDER is a regenerated date eg. 2014/07/31/14-16-34 which is used as a path.

Here is the python script:

#!/usr/bin/python

import boto, sys
from boto.s3.connection import S3Connection

access_key = 'accesskey'
secret_key = 'secretkey'
bucket_name = 'a name'
folder_path =  str(sys.argv[1]) if len(sys.argv) > 1 else ''

print("Forwarded folder path " + folder_path)

conn = S3Connection(access_key, secret_key)
bucket = conn.get_bucket(bucket_name)

print("Bucket Location:" + bucket.get_location())

for key in bucket.list(prefix=folder_path, delimiter=''):
        if '.' in key.name:
                file_name = key.name[len(folder_path)+1:]
                print("Downloading file " + file_name)
                key.get_contents_to_filename('/data/temp/' + file_name)

When I execute the bash script without changing the python /data/pythonscript.py $FOLDER line, I get the following output:

Forwarded folder path 2014/07/31/14-16-34 
Buckect Location: 

But when I change it to python /data/pythonscript.py 2014/07/31/14-16-34 , everything works:

Forwarded folder path 2014/07/31/14-16-34
Bucket Location: 
Downloading 2014/07/31/14-16-34/FacetedSearch.zip 
Downloading file FacetedSearch.zip
Downloading 2014/07/31/14-16-34/Location.zip
Downloading file Location.zip
Downloading 2014/07/31/14-16-34/LocationPage.zip
Downloading file LocationPage.zip

Best Answer

Perhaps the aws bash command is returning non-printable characters that you don't see with print(). Try removing them with tr:

FOLDER=$(./aws get $BUCKET"/location.txt" | tr -cd "[:print:]")
Related Question