Compare timestamp from two log files and report final time in minutes

text processingtimestamps

I have 2 Files with file-path and timestamps. I'm trying to compare contents from file1: "192.168.1.4_time-final2" with file2: "192.168.1.1_file-stat" and after successful File-path match subtract respective timestamp and get result in minutes.

cat 192.168.1.1_file-stat

Mon 19 Jul 2021 03:52:09 PM IST /etc/nginx/nginx.conf
Mon 19 Jul 2021 03:52:09 PM IST /home/fes/nginxproxy.conf
Wed 03 Mar 2021 03:33:54 PM IST /home/fes/nginx/createfile
Mon 19 Jul 2021 08:52:15 PM IST /home/fes/nginx/Templates
Wed 03 Mar 2021 03:33:54 PM IST /home/fes/nginx/Templates/testPortalSAML.default
Mon 19 Jul 2021 07:28:05 PM IST /home/fes/nginx/Templates/Default.default
Wed 03 Mar 2021 03:33:54 PM IST /home/fes/nginx/Templates/testPortal.default
Mon 19 Jul 2021 08:52:15 PM IST /home/fes/nginx/Templates/BT.default

cat 192.168.1.4_time-final2

Thu 22 Jul 2021 12:46:39 PM IST /home/fes/nginx/createfile
Thu 22 Jul 2021 12:47:24 PM IST /home/fes/nginx/Templates/testPortal.default

Best Answer

You may convert the file from the format that you have to a tab-delimited form where the first column is the Unix timestamp and the second is the pathname.

Assuming GNU date and a shell like bash or zsh that has process substitutions and $'...' strings (used later):

paste <( date -f <( cut -d ' ' -f -7 file ) +%s ) \
      <( cut -d ' ' -f 8- file )

This makes GNU date convert the timestamps in the seven first space-delimited columns of the original file to Unix timestamps. It then pastes these together with the pathnames.

The output for the first file:

1626690129      /etc/nginx/nginx.conf
1626690129      /home/fes/nginxproxy.conf
1614765834      /home/fes/nginx/createfile
1626708135      /home/fes/nginx/Templates
1614765834      /home/fes/nginx/Templates/testPortalSAML.default
1626703085      /home/fes/nginx/Templates/Default.default
1614765834      /home/fes/nginx/Templates/testPortal.default
1626708135      /home/fes/nginx/Templates/BT.default

and for the second file:

1626938199      /home/fes/nginx/createfile
1626938244      /home/fes/nginx/Templates/testPortal.default

We may then sort the files on the second column and use join to extract the ones with the same second column:

join -t $'\t' -1 2 -2 2 \
        <( paste <( date -f <( cut -d ' ' -f -7 file1 ) +%s ) \
                <( cut -d ' ' -f 8- file1 ) | sort -k 2,2 ) \
        <( paste <( date -f <( cut -d ' ' -f -7 file2 ) +%s ) \
                <( cut -d ' ' -f 8- file2 ) | sort -k 2,2 )

The output given your example input files:

/home/fes/nginx/Templates/testPortal.default    1614765834      1626938244
/home/fes/nginx/createfile      1614765834      1626938199

This may then be passed through a simple awk program to compute the time difference in minutes:

join -t $'\t' -1 2 -2 2 \
        <( paste <( date -f <( cut -d ' ' -f -7 file1 ) +%s ) \
                <( cut -d ' ' -f 8- file1 ) | sort -k 2,2 ) \
        <( paste <( date -f <( cut -d ' ' -f -7 file2 ) +%s ) \
                <( cut -d ' ' -f 8- file2 ) | sort -k 2,2 ) |
awk -F '\t' '{ print ($NF - $(NF-1))/60, $1 }'

The final output then:

202874 /home/fes/nginx/Templates/testPortal.default
202873 /home/fes/nginx/createfile
Related Question