Bash – scp files from particular folder in parallel

bashgnu-parallellinuxrsyncscp

I want to scp files from machineA into my machineB and this is how I am doing it. I am copying bunch of files one by one from primary folder of machineA to primary folder of machineB and secondary folder of machineA to secondary folder of machineB.

trinity@machineB:~$ scp trinity@machineA:/data01/primary/* /data01/primary/
trinity@machineB:~$ scp trinity@machineA:/data02/secondary/* /data02/secondary/

Is there any way by which I can copy multiple files in parallel? Like five files at a time from a folder? So instead of copying one files at a time, I want to copy five files from primary or secondary folders respectively?

Basically I want to copy whatever is there in primary and secondary folders of machineA into machineB parallely.

I also have GNU Parallel installed on my box if I can use that. I tried below command but it doesn't work. I was expecting that it should copy 5 files in parallel at a time until everything gets copied from that folder.

parallel -j 5 scp trinity@machineA:/data01/primary/* /data01/primary/

Anything wrong with my parallel syntax? What is the best way by which I can copy five files in parallel from a remote folder until everything gets copied from it?

Best Answer

You need the * expansion to happen on the remote side:

ssh machineA 'parallel -j 5 scp {} machineB:/data01/primary/ ::: /data01/primary/*'

Related Solutions

Bash – How to launch two threads in bash shell script

The obvious is:

parallel -j 2 do_CopyInPrimary ::: "${PRIMARY_PARTITION[@]}" &
parallel -j 2 do_CopyInSecondary ::: "${SECONDARY_PARTITION[@]}" &
wait

But this way the secondary does not wait for the primary to finish and it does not check if the primary was successful. Let us assume that $PRIMARY_PARTITION[1] corresponds to $SECONDARY_PARTITION[1] (so if you cannot read the file from $PRIMARY_PARTITION[1] you will read it from $SECONDARY_PARTITION[1] - That also means that $PRIMARY_PARTITION and $SECONDARY_PARTITION have the same number of elements). Then you can condition the running of $SECONDARY_PARTITION[1] on $PRIMARY_PARTITION[1].

do_Copy() {
  PRIMARY_PARTITION=(550 274 2 546 278) # this will have more file numbers
  SECONDARY_PARTITION=(1643 1103 1372 1096 1369) # this will have more file numbers
  pel=${PRIMARY_PARTITION[$1]}
  sel=${SECONDARY_PARTITION[$1]}
  do_CopyInPrimary $pel || 
    do_CopyInSecondary $sel || 
    echo Could not copy neither $pel nor $sel
}
export -f do_Copy
# Number of elements in PRIMARY_PARTITION == SECONDARY_PARTITION
seq ${#PRIMARY_PARTITION[@]} | parallel -j 2 do_Copy

This will get the dependency right, but it will only copy 2 at a time in total. With -j4 you risk running 4 primaries at the same time, so we need to guard against that, too:

do_Copy() {
  PRIMARY_PARTITION=(550 274 2 546 278) # this will have more file numbers
  SECONDARY_PARTITION=(1643 1103 1372 1096 1369) # this will have more file numbers
  pel=${PRIMARY_PARTITION[$1]}
  sel=${SECONDARY_PARTITION[$1]}
  sem -j2 --fg --id primary do_CopyInPrimary $pel || 
    sem -j2 --fg --id secondary do_CopyInSecondary $sel || 
    echo Could not copy neither $pel nor $sel
}
export -f do_Copy
# Number of elements in PRIMARY_PARTITION == SECONDARY_PARTITION
seq ${#PRIMARY_PARTITION[@]} | parallel -j 4 do_Copy

sem will limit the number of primaries to 2 and the number of secondaries to 2.

Related Question