Linux – crop an image without fully loading it into memory

decodingimage editinglinuxmemorypng

I managed to generate a PNG image with a fractal.

The image is 65,536 pixels high and 65,536 pixels wide. It’s too big to be decoded into memory and displayed. It has probably a lot of unused space near the borders and I want to trim it to significantly reduce its area.

I tried GIMP and GraphicsMagick, but GIMP froze my computer and GraphicsMagick failed to allocate enough memory for the image.

I use Linux and I have 16GB of RAM. The compressed image is 6.2MB in size.

Can I trim the borders without loading the image fully into memory?

FWIW, in the end I used my school's server but I still want to know the answer.

Best Answer

You already know the cartesian definition of the image.

Consider some Python code to read the file in reasonably-sized buffer-loads - choose one or multiple lines of image pixels per buffer, but sufficiently few with regard to memory-availability.

On most pixel-lines, the left-most and right-most "border" pixels can be discarded. The first few, and last few, "border" lines, similarly.

The buffer-size of the output file can, and will, be different to that of the input file, by definition.

Related Solutions

Windows – Pipe file content into PowerShell command without loading the entire file to memory

It seems that PowerShell Get-Content is not very efficient when it comes to huge files, so you may be able to use the -ReadCount switch to tell it how many lines to pipe at a time from the file. I put 500 below so it'd pipe 500 lines at a time.

Read the Microsoft PowerShell article below too.

https://technet.microsoft.com/en-us/library/hh849787.aspx

PowerShell v 5.0 Example

Get-Content -ReadCount 500 backup.sql | & psql --username=... db_name

PowerShell Legacy Version Example

Get-Content -Read 500 backup.sql | & psql --username=... db_name

Since PowerShell Get-Content isn't effecient with huge files, have a look at Start-Process

Read the Microsoft PowerShell article below too.

https://technet.microsoft.com/en-us/library/hh849848.aspx

PowerShell v 5.0 Example (using -RedirectStandardInput switch)

Start-Process "C:\Program Files\PostgreSQL\<version>\bin\psql.exe" '--username=... db_name' -RedirectStandardInput backup.sql -NoNewWindow -Wait

Windows – Reducing Windows 10 x64 memory footprint – can I tune the system’s automatic memory compression

You can try adding more memory pressure to your system such that Windows pushes memory pages from RAM to the page file. The sysinternals testlimit program lets you allocate lots of memory (as outlined in one of Russinovich's blog posts), which should (hopefully) convince the memory manager to push data to the page file.

Run the following and let it hammer for a while (monitor your RAM in task manager), then terminate it.

testlimit64 -m 100

You should see other processes physical commit charge (ie: actual RAM usage) decrease and testlimit hog everything. When testlimit ends, you should have more RAM available than when you started.

Important note: this may let you start your VM, but will also slow down all the other programs running on your computer because when you next use them, they'll have less physical RAM to use and will need to load from the page file.

(This is a workaround rather than an actual fix. More RAM is the best solution, as already noted) .

Best Answer

Related Solutions

Windows – Pipe file content into PowerShell command without loading the entire file to memory

Windows – Reducing Windows 10 x64 memory footprint – can I tune the system’s automatic memory compression

Related Question