Shell – How to cat files together, adding missing newlines at end of some files

catnewlinesshelltext processing

I have a bunch of .text files, most of which end with the standard nl.

A couple don't have any terminator at end. The last physical byte is (generally) an alphameric character.

I was using cat *.text >| /tmp/joined.text, but then noticed a couple of places in joined.text where the first line of a file appeared at the end of the last line of a previous file. Inspecting the previous file, I saw there wasn't a line terminator — concatenation explained.

That raised the question, what's the easiest way to concatenate, sticking in the missing newline? What about these options?

A solution that might effectively add a blank line to some input files. For me, that's not a problem as the processing of joined.text can handle it.
A solution that adds the cr/fl only to files that do not already end that way.

Best Answer

Another command that can add newlines if needed is awk, so:

awk 1 ./*.txt

The 1 here is the simplest way to get a true condition in awk, which works for this purpose since awk default action on true conditions is to print the input lines.

Related Solutions

How to search for text in a file ignoring newlines

The GNU grep can do it

grep -z 'is\san\sexample\sfile.' file

To fulfill some points which arise in comments there are some modifications to script:

 grep -oz '^[^\n]*\bis\s*an\s*example\s*file\.[^\n]*' file

Regarding huge files I have no imagination of memory limitation but in the case of problem you are free to use sed

sed '/\bis\b/{
          :1
          N
          /file\.\|\(\n.*\)\{3\}/!b1
         }
     /\<is\s*an\s*example\s*file\./p
     D' file

that keep no more than 4-lines (because 4 words in pattern) in memory ($\n.*$\{3\}).

Shell – How to remove the newline from the last line in a file in order to add text to that line

If all you want to do is add text to the last line, it's very easy with sed. Replace $ (pattern matching at the end of the line) by the text you want to add, only on lines in the range $ (which means the last line).

sed '$ s/$/ and Linux/' <file >file.new &&
mv file.new file

which on Linux can be shortened to

sed -i '$ s/$/ and Linux/' file

If you want to remove the last byte in a file, Linux (more precisely GNU coreutils) offers the truncate command, which makes this very easy.

truncate -s -1 file

A POSIX way to do it is with dd. First determine the file length, then truncate it to one byte less.

length=$(wc -c <file)
dd if=/dev/null of=file obs="$((length-1))" seek=1

Note that both of these unconditionally truncate the last byte of the file. You may want to check that it's a newline first:

length=$(wc -c <file)
if [ "$length" -ne 0 ] && [ -z "$(tail -c -1 <file)" ]; then
  # The file ends with a newline or null
  dd if=/dev/null of=file obs="$((length-1))" seek=1
fi

Best Answer

Related Solutions

How to search for text in a file ignoring newlines

Shell – How to remove the newline from the last line in a file in order to add text to that line

Related Question