Shell – Remove all duplicate word from string using shell script

duplicateshellshell-scripttext processingxargs

I have a string like

"aaa,aaa,aaa,bbb,bbb,ccc,bbb,ccc"

I want to remove duplicate word from string then output will be like

"aaa,bbb,ccc"

I tried This code Source

$ echo "zebra ant spider spider ant zebra ant" | xargs -n1 | sort -u | xargs

It is working fine with same value,but when I give my variable value then it is showing all duplicate word also.

How can I remove duplicate value.

UPDATE

My question is adding all corresponding value into a single string if user is same .I have data like this ->

   user name    | colour
    AAA         | red
    AAA         | black
    BBB         | red
    BBB         | blue
    AAA         | blue
    AAA         | red
    CCC         | red
    CCC         | red
    AAA         | green
    AAA         | red
    AAA         | black
    BBB         | red
    BBB         | blue
    AAA         | blue
    AAA         | red
    CCC         | red
    CCC         | red
    AAA         | green

In coding I fetch all distinct user then I concatenate color string successfully .For that I am using code –

while read the records 

    if [ "$c" == "" ]; then  #$c I defined global
        c="$colour1"
    else
        c="$c,$colour1" 
    fi

When I print this $c variable i get the output (For User AAA)

"red,black,blue,red,green,red,black,blue,red,green,"

I want to remove duplicate color .Then desired output should be like

"red,black,blue,green"

For this desired output i used above code

 echo "zebra ant spider spider ant zebra ant" | xargs -n1 | sort -u | xargs

but it is displaying the output with duplicate values .Like

"red,black,blue,red,green,red,black,blue,red,green,"
Thanks

Best Answer

One more awk, just for fun:

$ a="aaa bbb aaa bbb ccc aaa ddd bbb ccc"
$ echo "$a" | awk '{for (i=1;i<=NF;i++) if (!a[$i]++) printf("%s%s",$i,FS)}{printf("\n")}'
aaa bbb ccc ddd

By the way, even your solution works fine with variables:

$ b="zebra ant spider spider ant zebra ant" 
$ echo "$b" | xargs -n1 | sort -u | xargs
ant spider zebra

Related Solutions

Bash – How to remove duplicate files using bash

I'm working on Linux, which means the is the command md5sum which outputs:

> md5sum *
d41d8cd98f00b204e9800998ecf8427e  file_1
d41d8cd98f00b204e9800998ecf8427e  file_10
d41d8cd98f00b204e9800998ecf8427e  file_2
d41d8cd98f00b204e9800998ecf8427e  file_3
d41d8cd98f00b204e9800998ecf8427e  file_4
d41d8cd98f00b204e9800998ecf8427e  file_5
d41d8cd98f00b204e9800998ecf8427e  file_6
d41d8cd98f00b204e9800998ecf8427e  file_7
d41d8cd98f00b204e9800998ecf8427e  file_8
d41d8cd98f00b204e9800998ecf8427e  file_9
b026324c6904b2a9cb4b88d6d61c81d1  other_file_1
31d30eea8d0968d6458e0ad0027c9f80  other_file_10
26ab0db90d72e28ad0ba1e22ee510510  other_file_2
6d7fce9fee471194aa8b5b6e47267f03  other_file_3
48a24b70a0b376535542b996af517398  other_file_4
1dcca23355272056f04fe8bf20edfce0  other_file_5
9ae0ea9e3c9c6e1b9b6252c8395efdc1  other_file_6
84bc3da1b3e33a18e8d5e1bdd7a18d7a  other_file_7
c30f7472766d25af1dc80b3ffc9a58c7  other_file_8
7c5aba41f53293b712fd86d08ed5b36e  other_file_9

Now using awk and xargs the command would be:

md5sum * | \
sort | \
awk 'BEGIN{lasthash = ""} $1 == lasthash {print $2} {lasthash = $1}' | \
xargs rm

The awk part initializes lasthash with the empty string, which will not match any hash, and then checks for each line if the hash in lasthash is the same as the hash (first column) of the current file (second column). If it is, it prints it out. At the end of every step it will set lasthash to the hash of the current file (you could limit this to only be set if the hashes are different, but that should be a minor thing especially if you do not have many matching files). The filenames awk spits out are fed to rm with xargs, which basically calls rm with what the awk part gives us.

You probably need to filter directories before md5sum *.

Edit:

Using Marcins method you could also use this one:

comm -1 -2 \
  <(ls) | \
  <(md5sum * | \
    sort -k1 | \
    uniq -w 32 | \
    awk '{print $2}' | \
    sort) \
xargs rm

This substracts from the filelist optained by ls the first filename of each unique hash optained by md5sum * | sort -k1 | uniq -w 32 | awk '{print $2}'.

Bash – How to decode a list of base64-encoded file names

Just add the base64 encoding of newline (Cg==) after each file name and pipe the whole thing to base64 -d:

find . -name "*_*" -printf "%f\n" |
  sed -n 's/_....-..-..\.pdf$/Cg==/p' |
  base64 -d

With your approach, that would have to be something like:

find . -name "*_*" -printf "%f\0" |
  sed -zn 's/_....-..-..\.pdf$//p' |
  xargs -r0 sh -c '
    for i do
      echo "$i" | base64 -d
    done' sh

as you need a shell to create those pipelines. But that would mean running several commands per file which would be quite inefficient.

Best Answer

Related Solutions

Bash – How to remove duplicate files using bash

Bash – How to decode a list of base64-encoded file names

Related Question