MacOS – Determine which app is opening too many files

crashfilesystemmacosperformanceresources

Sometimes my Mac starts behaving weirdly, a typical symptom being that I can't open certain apps that were working just fine recently. This just happened and I traced this issue to an excess of open files in the system. For instance:

$ man lsof
sh: pipe error: Too many open files in system
Error executing formatting or display command.
System command (cd '/usr/share/man' && (echo ".ll 24.1i"; echo ".nr LL 24.1i"; /bin/cat '/usr/share/man/man8/lsof.8') | /usr/bin/tbl | /usr/bin/groff -Wall -mtty-char -Tascii -mandoc -c | (/usr/bin/less -is || true)) exited with status 33024.
Man page /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/usr/share/man/man8/lsof.8 is identical to /usr/share/man/man8/lsof.8
No manual entry for lsof

Also:

$ /Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome 
dlopen /Applications/Google Chrome.app/Contents/Frameworks/Google Chrome Framework.framework/Versions/76.0.3809.100/Helpers/Google Chrome Helper.app/Contents/MacOS/../../../../Google Chrome Framework: dlopen(/Applications/Google Chrome.app/Contents/Frameworks/Google Chrome Framework.framework/Versions/76.0.3809.100/Helpers/Google Chrome Helper.app/Contents/MacOS/../../../../Google Chrome Framework, 261): no suitable image found.  Did find:
    /Applications/Google Chrome.app/Contents/Frameworks/Google Chrome Framework.framework/Versions/76.0.3809.100/Helpers/Google Chrome Helper.app/Contents/MacOS/../../../../Google Chrome Framework: open() failed with errno=23
    /Applications/Google Chrome.app/Contents/Frameworks/Google Chrome Framework.framework/Versions/76.0.3809.100/Helpers/Google Chrome Helper.app/Contents/MacOS/../../../../Google Chrome Framework: stat() failed with errno=3
    /Applications/Google Chrome.app/Contents/Frameworks/Google Chrome Framework.framework/Versions/76.0.3809.100/Google Chrome Framework: open() failed with errno=23
[0822/142149.543101:ERROR:mac_util.cc(204)] _CFCopySystemVersionDictionary failed
[0822/142149.543447:WARNING:process_memory_mac.cc(93)] mach_vm_read(0x7ffee8237000, 0x2000): (os/kern) invalid address (1)
[92085:775:0822/112149.554485:FATAL:process_singleton_posix.cc(236)] : Too many open files in system (23)
[0822/142149.557563:ERROR:file_io_posix.cc(140)] open /Users/decio/Library/Application Support/Google/Chrome/Crashpad/settings.dat: Too many open files in system (23)
[0822/142149.557717:ERROR:file_io_posix.cc(148)] open /Users/decio/Library/Application Support/Google/Chrome/Crashpad/new/85472cbe-1c22-4598-94bf-a964bf67b0c5.dmp: Too many open files in system (23)
[0822/142149.558108:ERROR:mac_util.cc(204)] _CFCopySystemVersionDictionary failed
[0822/112149.558599:WARNING:process_memory_mac.cc(93)] mach_vm_read(0x7ffeef5d5000, 0x2000): (os/kern) invalid address (1)
[92089:775:0822/112149.643059:ERROR:resource_bundle.cc(803)] Failed to load /Applications/Google Chrome.app/Contents/Frameworks/Google Chrome Framework.framework/Versions/76.0.3809.100/Resources/chrome_100_percent.pak
Some features may not be available.
[92089:775:0822/112149.643114:ERROR:resource_bundle.cc(803)] Failed to load /Applications/Google Chrome.app/Contents/Frameworks/Google Chrome Framework.framework/Versions/76.0.3809.100/Resources/chrome_200_percent.pak
Some features may not be available.
[92089:775:0822/112149.643192:ERROR:resource_bundle.cc(363)] failed to load locale.pak
[92089:775:0822/112149.643221:ERROR:resource_bundle.cc(803)] Failed to load /Applications/Google Chrome.app/Contents/Frameworks/Google Chrome Framework.framework/Versions/76.0.3809.100/Resources/resources.pak
Some features may not be available.
[0822/112149.648315:ERROR:file_io_posix.cc(140)] open /Users/decio/Library/Application Support/Google/Chrome/Crashpad/settings.dat: Too many open files in system (23)
[0822/112149.648369:ERROR:file_io_posix.cc(148)] open /Users/decio/Library/Application Support/Google/Chrome/Crashpad/new/f3454bf7-4dc1-4a39-9b23-112edbf9565b.dmp: Too many open files in system (23)
[0822/112149.649774:ERROR:mac_util.cc(204)] _CFCopySystemVersionDictionary failed
[0822/112149.649989:WARNING:process_memory_mac.cc(93)] mach_vm_read(0x7ffee426c000, 0x2000): (os/kern) invalid address (1)
Trace/BPT trap: 5
[0822/112149.746121:WARNING:system_snapshot_mac.cc(42)] sysctlbyname kern.nx: No such file or directory (2)

I restarted a few apps and managed to go back under the limit and run new apps again.

However, I'd like to be more systematic. For all I know, the few apps that I restarted were just barely enough to go under the limit, and I'll soon face the same issue. Rather than closing random apps which may or may not be contributing to this issue, I'd like to locate the specific app that is hoarding file handles and close it.

I thought lsof would help me pinpoint the problematic app, but manually parsing its default output is unfeasible. I could write a shell script or Python program to group file handles by app, but I'm thinking there may be a switch to lsof that does this grouping automatically, or maybe an app similar to Activity Monitor but which display the number of open file handles per process.

My question then is: how can I display the number of open file handles grouped per process?

I'm running macOS 10.14.5, if it makes a difference. Also, while I'm aware that I can increase the open file limit, I feel the real issue is that one of my apps is essentially leaking resources, not unlike a memory leak, and it's best for the performance of the system to just restart the affected app than blindly increase limits and allow the app to continue hoarding resources.

Best Answer

Time to bring the power of Unix pipes, sort, and awk to the rescue. But since lsof takes a long time to run I'll save the output and analyze that, rather than piping lsof directly into the pipe.

For reference here's the lsof output format:

[iMac:~] droot% head /tmp/a
COMMAND    PID  USER   FD      TYPE             DEVICE   SIZE/OFF     NODE NAME
loginwind   94 droot  cwd       DIR                1,7        896        2 /
loginwind   94 droot  txt       REG                1,7    1237152 18011104 /System/Library/CoreServices/loginwindow.app/Contents/MacOS/loginwindow
loginwind   94 droot  txt       REG                1,7      21024 18566026 /Library/Preferences/Logging/.plist-cache.HrVraTtH
loginwind   94 droot  txt       REG                1,7   27154336 18044520 /usr/share/icu/icudt62l.dat

And my analysis:

lsof > /tmp/a
cat /tmp/a | awk '{print $2}' | sort | uniq -c | sort -r -n | head

 511 724
 469 2790
 384 2791
 360 2793
 350 2798
 227 286
 224 658
 210 3833
 197 655
 190 698

Explaining the non-obvious lines:

  1. awk '{print $2}' prints the 2nd field in each line (the PID in this case). This is about all the awk I remember. Really useful!
  2. uniq -c Takes a sorted list of PIDs and eliminates duplicate adjacent lines, with the -c modifier it adds a count of the number of lines found (in our case that's one line per open file from that process).
  3. "sort -r -n" sorts numerically in reverse order (largest number at top)
  4. "head" prints out the first 10 lines (10 processes with largest number of open files).

That shows that PID 724 has 511 open files on my system. To look in more detail:

cat /tmp/a | grep " 724 "

In my case that process was Dropbox, which is "interesting" but I can see that that process might need many open files. So going to the next one:

cat /tmp/a | grep " 2790 "

And so on down the list. Hopefully your "offender" is obvious from the lsof output sorted by number of open files per PID.

For total number of open files use "wc" (wordcount):

[iMac:~] droot% cat /tmp/a | wc
9777   90500 1445558

Which means 9776 open files in my case (have to subtract the header line).

Related Question