Linux command to remove the duplicate lines but keep the first occurrence

All you should need to do is add "\" at the end of each line and it should be good to go.

So yours will look like:

tar -cvpzf /share/Recovery/Snapshots/$(hostname)_$(date +%Y%m%d).tar.gz \
    --exclude=/proc \
    --exclude=/lost+found \
    --exclude=/sys \
    --exclude=/mnt \
    --exclude=/media \ 
    --exclude=/dev \
    --exclude=/share/Archive \
    /

A Few Shortcuts

(based on your comment update for setting $HOSTNAME)

$HOSTNAME

Two options to set that:

1. Set HOSTNAME

   HOSTNAME=$(hostname)

2. Use command substitution (e.g. $(command))

   So it would look like above. That just makes the command run before using it.

$DATE

Another variable avoided would be easily:

$(hostname)_$(date +%Y%m%d).tar.gz \

$ man date will have the formats for the date options, the above is YYYYmmdd

The solutions posted by others do not work on my Debian Jessie: they keep a single copy of any duplicate line, while it is my understanding of the OP that all copies of the duplicate lines are to be kept. If I have understood the OP right, then ...

1. The following command

   awk '!seen[$0]++' file
   

   removes all duplicate lines.

2. The following command

   awk 'seen[$0]++' file 
   

   outputs all the duplicates, but not the original copy: i.e., if a line appears n times, it outputs the line n-1 times.

3. Then the command

   awk 'seen[$0]++' file > temp && awk '!seen[$0]++' file >> temp
   

   solves your problem. The lines are not in the original order.

4. If you want lines which have two or more duplicates, you can now iterate the above:

   awk 'seen[$0]++' file | awk 'seen[$0]++' > temp
   

   keeps n-2 copies of the lines which have n>1 duplicates. Now

   awk '!seen[$0]++' temp > temp1 
   

   removes all duplicate lines from the temp file, and you can now obtain what you wish (i.e. only the lines with n>1 duplicates) as follows:

   cat temp1 >> temp; cat temp1 >> temp
   

5. If you need to do this for lines which appear N or more times, the following command

     awk 'seen[$0]++ && seen[$0] > N' file 
   

   is simpler than chaining N times the command awk 'seen[$0]++' file.