Curl – Fix Issues with Curling List of Domains from a File

bashcurlrecursivevariable

I have a bash script below that is intended to curl against a list of domains, from a file, and output the results.

#!/bin/bash
    
baseurl=https://csp.infoblox.com
domains=/home/user/Documents/domainlist
B1Dossier=/tide/api/data/threats/state/host?host=$domains
APIKey=<REDACTED>
AUTH="Authorization: Token $APIKey"
        
        
for domain in $domains; do curl -H "$AUTH" -X GET ${baseurl}${B1Dossier} > /tmp/outputfile; done

Unfortunately, the script is not going through each domain in the file whatsoever.

To help understand, I have listed the expectation/explanation of the script:

  • Within the file, /home/user/Documents/domainlist, I have a handful of domains.
  • I'm attempting to use the API to check each domain in the file, by appending the variable $domains at the end of B1Dossier
  • The expectation is that it would run the specified curl command against each domain, within the file, and output the results.

For added visibility, I included the working curl command used for a single domain below:

curl -H 'Authorization: Token <REDACTED>' -X GET https://csp.infoblox.com/tide/api/data/threats/state/host?host=<place domain here>

Can someone assist in what I'm doing wrong and how I can fix this?

Best Answer

You can read the domains from file to an array and loop for them.

baseurl="https://csp.infoblox.com"
B1Dossier="/tide/api/data/threats/state/host?host="
url="${baseurl}${B1Dossier}"

# read domains to an array
mapfile -t domains < /home/user/Documents/domainlist

# loop for domains
for d in "${domains[@]}"; do
    curl -H "$AUTH" -X GET "${url}${d}" >> temp
done

Notes:

In your command, using B1Dossier into the loop, has no effect, it seems you were waiting some kind of a recursive evaluation, because the domain is contained in B1Dossier and you loop for domain. But your url doesn't change inside the loop this way.

Also, you have to append the responses to your destination file using >> or else every next response would overwrite the previous one.

Related Question