Bash – How to Check Command Success

%CPU should be low during a copy. The CPU tells the disk controller "grab data from sectors X–Y into memory buffer at Z". Then it goes and does something else (or sleep, if there is nothing else). The hardware triggers an interrupt when the data is in memory. Then the CPU has to copy it a few times, and tells the network card "transmit packets at memory locations A, B, and C". Then it goes back to doing something else.

You're pushing ~240mbps. On a gigabit LAN, you ought to be able to do at least 800mbps, but:

1. That's shared among everyone using the file server (and possibly a connection between switches, etc.)
2. That's limited by the speed the file server can handle the write, keeping in mind its disk I/O bandwidth is shared by everyone using it.
3. You didn't specify how you're accessing the file server (NFS, CIFS (Samba), AFS, etc.). You may need to tune your network mount, but on anything half-recent the defaults are usually pretty sane.

For tracking down the bottleneck, iostat -kx 10 is going to be a useful command. It'll show you the utilization on your local hard disks. If you can run that on the file server, it'll tell you how busy the file server is.

The general solution is going to be to speed up that bottleneck, which of course you don't have the budget for. But, there are a couple of special cases where you can find a faster approach:

• If the files are compressible, and you have a fast CPU, doing a minimal compress on-the-fly might be quicker. Something like lzop or maybe gzip --fastest.
• If you are only changing a few bits here and there, and then sending the file back, only sending deltas will be much faster. Unfortunately, rsync won't really help here, as it will need to read the file on both sides to find the delta. Instead, you need something that keeps track of the delta as you change the file... Most approaches here are app-specific. But its possible that you could rig something up with, e.g., device-mapper (see the brand new dm-era target) or btrfs.
• If you're copying the same data to multiple machines, you can use something like udpcast to send it to all the machines at once.

And, since you note you're not the sysadmin, I'm guessing that means you have a sysadmin. Or at least someone responsible for the file server & network. You should probably ask him/her/them, they should be much more familiar with the specifics of your setup. Your sysadmin(s) should at least be able to tell you what transfer rate you can reasonably expect.

The obvious is:

parallel -j 2 do_CopyInPrimary ::: "${PRIMARY_PARTITION[@]}" &
parallel -j 2 do_CopyInSecondary ::: "${SECONDARY_PARTITION[@]}" &
wait

But this way the secondary does not wait for the primary to finish and it does not check if the primary was successful. Let us assume that $PRIMARY_PARTITION[1] corresponds to $SECONDARY_PARTITION[1] (so if you cannot read the file from $PRIMARY_PARTITION[1] you will read it from $SECONDARY_PARTITION[1] - That also means that $PRIMARY_PARTITION and $SECONDARY_PARTITION have the same number of elements). Then you can condition the running of $SECONDARY_PARTITION[1] on $PRIMARY_PARTITION[1].

do_Copy() {
  PRIMARY_PARTITION=(550 274 2 546 278) # this will have more file numbers
  SECONDARY_PARTITION=(1643 1103 1372 1096 1369) # this will have more file numbers
  pel=${PRIMARY_PARTITION[$1]}
  sel=${SECONDARY_PARTITION[$1]}
  do_CopyInPrimary $pel || 
    do_CopyInSecondary $sel || 
    echo Could not copy neither $pel nor $sel
}
export -f do_Copy
# Number of elements in PRIMARY_PARTITION == SECONDARY_PARTITION
seq ${#PRIMARY_PARTITION[@]} | parallel -j 2 do_Copy

This will get the dependency right, but it will only copy 2 at a time in total. With -j4 you risk running 4 primaries at the same time, so we need to guard against that, too:

do_Copy() {
  PRIMARY_PARTITION=(550 274 2 546 278) # this will have more file numbers
  SECONDARY_PARTITION=(1643 1103 1372 1096 1369) # this will have more file numbers
  pel=${PRIMARY_PARTITION[$1]}
  sel=${SECONDARY_PARTITION[$1]}
  sem -j2 --fg --id primary do_CopyInPrimary $pel || 
    sem -j2 --fg --id secondary do_CopyInSecondary $sel || 
    echo Could not copy neither $pel nor $sel
}
export -f do_Copy
# Number of elements in PRIMARY_PARTITION == SECONDARY_PARTITION
seq ${#PRIMARY_PARTITION[@]} | parallel -j 4 do_Copy

sem will limit the number of primaries to 2 and the number of secondaries to 2.