Shell Script – Adding Lines to the Beginning and End of a Huge File

filesshell-script

I have the scenario where lines to be added on begining and end of the huge files.

I have tried as shown below.

  • for the first line: 

    sed -i '1i\'"$FirstLine" $Filename

  • for the last line: 

    sed -i '$ a\'"$Lastline" $Filename

But the issue with this command is that it is appending the first line of the file and traversing entire file. For the last line it's again traversing the entire file and appending a last line. Since its very huge file (14GB) this is taking very long time.

How can I add a line to the beginning and another to the end of a file while only reading the file once?

Best Answer

sed -i uses tempfiles as an implementation detail, which is what you are experiencing; however, prepending data to the beginning of a data stream without overwriting the existing contents requires rewriting the file, there's no way to get around that, even when avoiding sed -i.

If rewriting the file is not an option, you might consider manipulating it when it is read, for example:

{ echo some prepended text ; cat file ; } | command

Also, sed is for editing streams -- a file is not a stream. Use a program that is meant for this purpose, like ed or ex. The -i option to sed is not only not portable, it will also break any symlinks to your file, since it essentially deletes it and recreates it, which is pointless.

You can do this in a single command with ed like so:

ed -s file << 'EOF'
0a
prepend these lines
to the beginning
.
$a
append these lines
to the end
.
w
EOF

Note that depending on your implementation of ed, it may use a paging file, requiring you to have at least that much space available.

Related Question