MacOS – Sudden barrage of “too many open files” in log, cascading app failures

adobemacos

I came into work today with a cascade of "Application Unexpectedly Quit" messages on my desktop.

Looking in system.log I'm seeing many failures due to too many open files. The first instance looks like:

Core Sync[525]: dnssd_clientstub deliver_request: socketpair failed 24 (Too many open files)
Core Sync[525]: tcp_connection_host_start 7318 DNSServiceGetAddrInfo failed: -65537

There's about a dozen or so of these pairs of errors, after which I see:

Core Sync[525]: nw_interface_create_with_index_and_name information socket creation failed: [24] Too many open files, dumping backtrace:
            [x86_64] libnetcore-583.50.1
        0   libsystem_network.dylib             0x00007fff967aade9 __nw_create_backtrace_string + 123
        1   libsystem_network.dylib             0x00007fff967ccc0a nw_interface_create_with_index_and_name + 657
        2   libsystem_network.dylib             0x00007fff967cd30f nw_interface_create_with_index + 195
        3   libsystem_network.dylib             0x00007fff967877fa tcp_connection_create_connected_path_evaluator_for_connected_dst + 182
        4   libsystem_network.dylib             0x00007fff96787499 tcp_connection_handle_destination_complete + 726
        5   libsystem_network.dylib             0x00007fff967bc686 __tcp_connection_destination_setup_socket_events_block_invoke_2 + 2658
        6   libdispatch.dylib                   0x00007fff87d4240b _dispatch_client_callout + 8
        7   libdispatch.dylib                   0x00007fff87d52675 _dispatch_source_latch_and_call + 2235
        8   libdispatch.dylib                   0x00007fff87d46a83 _dispatch_source_invoke + 983
        9   libdispatch.dylib                   0x00007fff87d47200 _dispatch_queue_drain + 1207
        10  libdispatch.dylib                   0x00007fff87d4d707 _dispatch_queue_invoke + 549
        11  libdispatch.dylib                   0x00007fff87d45d53 _dispatch_root_queue_drain + 538
        12  libdispatch.dylib                   0x00007fff87d45b00 _dispatch_worker_thread3 + 91
        13  libsystem_pthread.dylib             0x00007fff88a0c4de _pthread_wqthread + 1129
        14  libsystem_pthread.dylib             0x00007fff88a0a341 start_wqthread + 13

There's several screens full of this error and the previous one.

Following that is a bunch of other sorts of failure from a variety of applications, and an ongoing smattering of the above errors.

There's also several pages of kernel[0]: Sandbox: mdworker(802) deny(1) file-read-data /Applications/App Store.app
which I think is from Spotlight, probably a side-effect of the too many open files problem.

Using lsof and an awk script, I found that the process with the most open files is Core Sync with a pretty consistent count of 7227 files open.

CPU and memory usage seem to be what I'd expect.

I tried rebooting in safe mode, repairing the boot drive, resetting PRAM, and used Onyx to clear all the caches. I don't recall installing anything new or unusual recently, and everything was fine when I left work last night.

I'm on Mac OS 10.11.6 on a Mid 2012 Mac Pro.

Can anyone help?

Thanks!

Best Answer

So the problem turned out to be in Adobe CC. I ran the Adobe CC UNinstaller, and selected "Repair", rebooted, and things were back to whatever normal is. I just wish I'd connected Core Sync with Adobe sooner!

Oh, and in case anyone is interested, here's the script I used to figure what had the most files opened. It lists all the processes by name, and sorts by number of open files.

#!/bin/bash lsof +c 0 | gawk ' { if (! $1 in count) count[$1] = 0 count[$1]++ } END { for (var in count) print var, count[var] } ' | sort -n --key=2

It depends on GNU awk (gawk) so not sure how well it would work with the installed awk.

Related Question