How to Edit Large Files in Place – Best Tools and Methods

editorsfileshp-uxlarge files

I have a few files sized > 1 GB each. I need to remove last few bytes from the files. How can I do it? I prefer to edit file in place to save disk space.

I am on HP-UX.

Best Answer

Try using hexedit I haven't tried it on HP-UX but it should work. It allows you to move to a location in a file and truncate. I'm pretty sure that it does not read the whole file in but just seeks to the appropriate location for display.

Usage is fairly simple once you have launched it the arrow keys allow you to move around. F1 gives help. Ctrl-G moves to a location in the file (hint: to move to end use the size of the file from the bottom row of the display). Position the cursor on the first byte that you want to truncate and then press Escape T once you confirm the truncate will have been done. Ctrl-x exits.

Related Solutions

Merge nonzero blocks of huge (sparse) file A into huge file B

Your algorithm is implemented in GNU dd.

dd bs=512 if=A of=B conv=sparse,notrunc

Please verify this beforehand with some test files of your choice. You don't want to inadvertently damage your File B. A better algorithm would be to check whether B also has zeroes at that position, alas that's something dd does not do.

As for two different computers, you have several options. Use a network filesystem that supports seeks on writes (not all do); transfer the file beforehand; or pipe through SSH like so:

dd if=A | ssh -C B-host dd of=B conv=sparse,notrunc
# or the other way around
ssh -C A-host dd if=A | dd of=B conv=sparse,notrunc

The ssh -C option enables compression, you'd be transferring gigabytes of zeroes over the network otherwise.

Deduplication of lines in a large file

It seems to me that the process you're following at the moment is this, which fails with your out of memory error:

Create several data files
Concatenate them together
Sort the result, discarding duplicate records (rows)

I think you should be able to perform the following process instead

Create several data files
Sort each one independently, discarding its duplicates (sort -u)
Merge the resulting set of sorted data files, discarding duplicates (sort -m -u)

Best Answer

Related Solutions

Merge nonzero blocks of huge (sparse) file A into huge file B

Deduplication of lines in a large file

Related Question