Linux – How to cat all the log files within a range of dates

catcsvlinux

I have log files which are named in the following manner:

localhost_log_file.2017-09-01.txt
localhost_log_file.2017-09-02.txt
....

localhost_log_file.2017-10-30.txt

In other words, each log file has the following form:

localhost_log_file.YYYY-MM-DD.txt

I want to cat all the log files taken between the dates of 2017-09-03 and 2017-10-08, i.e. every log file starting from localhost_log_file.2017-09-03.txt through localhost_log_file.2017-10-08.txt.

Currently what I do is produce three intermediate files by separately executing each of the following commands:

for((i=3;i<=9;i++)) do cat localhost_log_file.2017-09-0$i.txt >> log1.csv ; done;

for((i=10;i<=30;i++)) do cat localhost_log_file.2017-09-$i.txt >> log2.csv ; done;

for((i=1;i<=8;i++)) do cat localhost_log_file.2017-10-0$i.txt >> log3.csv ; done;

Then I combine the intermediate files as follows:

cat log1.csv log2.csv log3.csv> totallog.csv

Is there a better way to do this?

Best Answer

How about using the date utility to iterate through the range of dates you're interested in? Here is what that might look like for your example:

# Set the date counter to the start date
d=2017-09-03

# Iterate until we reach the end date (i.e. the date after the last date we want)
while [ "$d" != 2017-10-09 ]; do

    # cat each file
    cat "localhost_log_file.${d}.txt";

    # Increment the date counter
    d="$(date -I -d "$d + 1 day")";

done

See this for more information:

Alternatively, you can pass the results of the loop to the cat command instead of invoking cat in the body of the loop.

Here is what that could look like using command-substitution:

d=2017-09-03
cat $(while [ "$d" != 2017-10-09 ]; do
    echo "localhost_log_file.${d}.txt";
    d="$(date -I -d "$d + 1 day")";
done)

And here is the same thing using a pipe and xargs:

d=2017-09-03
while [ "$d" != 2017-10-09 ]; do
    echo "localhost_log_file.${d}.txt";
    d="$(date -I -d "$d + 1 day")";
done | xargs cat
Related Question