Ubuntu – Sed Script – Reversing names

You can do:

for f in *.conf; do
    base=$(basename "$f" '.conf') # gives '25' from '25.conf'
    sed -i.before "s/example/$base/g" "$f"
done

When using the -i switch to sed, you must be absolutely sure that your sed command works because -i changes the files in-place. This means the generated output will overwrite the input file. If your replacement command (s/…/…/) is wrong, you may end up with empty files and no backups. Hence, I used -i.before which will leave a *.before file with the original content.

General method

• You can split each file into a header and a second file with the data lines
• Then you can easily edit a header separately with your current sed command
• Finally you can concatenate the header and the file with the data lines.

Light-weight tools to manage huge files

• You can use head and tail to create a head file and a data file.

• You can use cat to concatenate the modified head file and the data file.

• Efficient way to print lines from a massive file using awk, sed, or something else?

• Another method is to use split

Test

• I tested with your header and a file with 1080000000 numbered lines (size 19 Gib), totally 1080000007 lines, and it worked, the output file (with 1080000004 lines) was written in 5 minutes in my old hp xw8400 workstation (including typing the command to start the shellscript).

  $ ls -lh --time-style=full-iso huge*
  -rw-r--r-- 1 sudodus sudodus 19G 2018-12-15 19:50:45.278328120 +0100 huge.in
  -rw-r--r-- 1 sudodus sudodus 19G 2018-12-15 19:55:46.808798456 +0100 huge.out
  

• The big write operations were between the system partition on an SSD and a data partition on an HDD.

Shellscript

You need enough free space in the file system where you have /tmp for the huge temporary 'data' file, more than 9 GB according to your original question.

$ LANG=C df -h /tmp
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1       106G   32G   69G  32% /

This may seem an awkward way to do things, but it works for huge files without crashing the tools. Maybe you must store the temporary 'data' file somewhere else, for example in an external drive (but it will probably be slower).

#!/bin/bash

# $1 : FCIDUMP file to convert from "new format" to "old format"

if [ $# -ne 2 ]
then
  echo "Syntaxis: $0 fcidumpfile oldstylefile " 1>&2
  echo "Example:  $0 file.in file.out" 1>&2
  exit 1
fi

if [ "$1" == "$2" ]
then
  echo "The names of the input file and output file must differ"
  exit 2
exit
fi

endheader="$(grep -m 1 -n '&END' "$1" | cut -d: -f1)"
if [ "$endheader" == "" ]
then
  echo "Bad input file: the end marker of the header was not found"
  exit 3
fi
#echo "endheader=$endheader"

< "$1" head -n "$endheader" > /tmp/header
#cat /tmp/header

if egrep '&FCI ([a-zA-Z2 ]*=[0-9 ]*,){2,}' /tmp/header  > /dev/null
then
  echo "The provided file is already in old FCIDUMP format." 1>&2
  exit 4
fi

# run sed inline on /tmp/header 
sed '
{
:a; N; $!ba
s/\(=[^,]*,\)\n/\1 /g
s/\(&FCI\)\n/\1 /
s/ORBSYM/\n&/g
s/&END/ISYM=1,\n\//
}' -i /tmp/header 

if [ $? -ne 0 ]
then
  echo "Failed to convert the header format in /tmp/header"
  exit 5
fi

< "$1" tail -n +$(($endheader+1)) > /tmp/tailer

if [ $? -ne 0 ]
then
  echo "Failed to create the 'data' file /tmp/tailer"
  exit 6
fi

#echo "---"
#cat /tmp/tailer
#echo "---"

cat /tmp/header /tmp/tailer > "$2"

exit 0