MacOS – Why does mdworker segfault when scanning files owned by root

crashmacossnow leopardspotlighttime-machine

While investigating a disk with ever-dwindling free space and a CPU pegged at >100%, I eventually determined that the problem was mdworker repeatedly segfaulting, keeping syslogd and CrashReporter very busy.

I tried rebuilding Spotlight's indexes in the usual ways: first, via the Privacy tab in System Preferences -> Spotlight, then via mdworker -i off / ; mdworker -E -i on /, and the same again but with an intervening rm -rf /.Spotlight-V100 and reboot for good measure; nothing seemed to solve the problem.

Using the Privacy tab to exclude pretty much everything except /Applications/, and then adding/removing this folder to force a re-scan, and I was able to determine that some files are being properly indexed (and show up in Spotlight results) but some are not; a bit more poking around with opensnoop -n mdworker reveals that when mdworker is started with UID 501, to scan Application files owned by me it works fine (and same for a few other UIDs who own files in /Applications/), but when it is started with UID 89 (_spotlight, according to dscl . -list /Users UniqueID) – presumably to scan files owned by root – it segfaults.

Here's an example entry from Console:

2015-07-16 13:53:25 com.apple.launchd[1]    (0x100101670.mach_init.mdworker[13276]) Job appears to have crashed: Segmentation fault
2015-07-16 13:53:25 com.apple.ReportCrash.Root[13274]   2015-07-16 13:53:25.326 ReportCrash[13274:341b] Saved crash report for mdworker[13276] version ??? (???) to /Library/Logs/DiagnosticReports/mdworker_2015-07-16-135325-1_localhost.crash

And here is an excerpt from the crash report (they're all pretty much identical):

Process:         mdworker [13276]
Path:            /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/Metadata.framework/Versions/A/Support/mdworker
Identifier:      mdworker
Version:         ??? (???)
Code Type:       X86-64 (Native)
Parent Process:  launchd [1]

Date/Time:       2015-07-16 13:53:25.085 +0100
OS Version:      Mac OS X 10.6.8 (10K549)
Report Version:  6

Exception Type:  EXC_BAD_ACCESS (SIGSEGV)
Exception Codes: KERN_INVALID_ADDRESS at 0x000000010f5d1062
Crashed Thread:  3

[...]

Thread 3 Crashed:
0   ...ple.CoreServices.CarbonCore  0x00007fff867e7f0b CSStoreGetUnit + 84
1   com.apple.LaunchServices        0x00007fff821721ab _LSContainerCheckState + 65
2   com.apple.LaunchServices        0x00007fff82188fea _LSCopyLibraryItemURLs + 419
3   mdworker                        0x0000000100004305 0x100000000 + 17157
4   mdworker                        0x0000000100004c22 0x100000000 + 19490
5   mdworker                        0x00000001000050f3 0x100000000 + 20723
6   mdworker                        0x0000000100009aa2 0x100000000 + 39586
7   libSystem.B.dylib               0x00007fff80b94fd6 _pthread_start + 331
8   libSystem.B.dylib               0x00007fff80b94e89 thread_start + 13

[...]

I am reasonably sure that this is not being caused by the contents of the files it attempting to scan, because should be scanning in /Applications/, and opensnoop does not report it touching any files there (in fact, the list of files opened for each crashing UID 89 instance is identical, AFAICT).

It is possible that this problem is related to issues I have been having with Time Machine, which started roughly around the same time: backupd also segfaults unexpectedly – not instantly upon startup, but in the process of mounting my NAS backup volume. Here's an excerpt from a backupd crash report:

Thread 5 Crashed:
0   ...ple.CoreServices.CarbonCore  0x00007fff867e7f0b CSStoreGetUnit + 84
1   com.apple.LaunchServices        0x00007fff8217f3fb _LSBundleFindWithNode + 544
2   com.apple.LaunchServices        0x00007fff82177bd1 _LSFindOrRegisterBundleNode + 219
3   com.apple.LaunchServices        0x00007fff82177a85 _LSCopyItemAttributeForRefInfoWithOptions + 201
4   com.apple.LaunchServices        0x00007fff821799cf prepareAttributeValueForKey(__CFURL const*, __FileCache*, __CFString const*, void const**, __CFError**) + 79
5   com.apple.LaunchServices        0x00007fff82179934 prepareDistinctLocalizedNameValue(__CFURL const*, __FileCache*, __CFError**) + 36
6   com.apple.LaunchServices        0x00007fff8217990b prepareLocalizedNameValue(__CFURL const*, __FileCache*, __CFError**) + 9
7   com.apple.LaunchServices        0x00007fff82179712 LSPropertyProviderPrepareValues(__CFURL const*, __FileCache*, __CFString const* const*, void const**, long, void const*, __CFError**) + 51
8   ...ple.CoreServices.CarbonCore  0x00007fff867f8dd4 prepareValuesForBitmap(__CFURL const*, __FileCache*, _FilePropertyBitmap*, __CFError**) + 264
9   ...ple.CoreServices.CarbonCore  0x00007fff867f78bd _FSURLCopyResourcePropertiesForKeys + 980
10  com.apple.CoreFoundation        0x00007fff897dc562 CFURLCopyResourcePropertiesForKeys + 98
11  com.apple.DesktopServices       0x00007fff833737af TCFURLInfo::FetchProperties(bool) + 91
12  com.apple.DesktopServices       0x00007fff8337358f TCFURLInfo::Initialize(__CFURL const*, bool, bool) + 183
13  com.apple.DesktopServices       0x00007fff833d1acd TCFURLInfo::Initialize(char const*, unsigned int) + 89
14  com.apple.DesktopServices       0x00007fff833d369c TCFURLInfo::CreateDirectory(TUString const&, TUniqueNamer*, __FSFileSecurity*, bool, TCountedPtr<TCFURLInfo>&) const + 464
15  com.apple.DesktopServices       0x00007fff833dc95a TCopyWriter::CreateNewDestinationItem() + 178
16  com.apple.DesktopServices       0x00007fff833dd136 TCopyWriter::CreateItem() + 1126
17  com.apple.DesktopServices       0x00007fff833dd41e TCopyWriter::Write() + 146
18  com.apple.DesktopServices       0x00007fff833dd6a6 TCopyWriter::WriteTaskProc(void*) + 72
19  ...ple.CoreServices.CarbonCore  0x00007fff867e40d1 PrivateMPEntryPoint + 63
20  libSystem.B.dylib               0x00007fff80b94fd6 _pthread_start + 331
21  libSystem.B.dylib               0x00007fff80b94e89 thread_start + 13

I have used Disk Utility to (live) verify the volume and repair permissions. I have tried re-installing the 10.6.8 Combo Update 1.1 and Supplemental update.

What might be causing these crashes, and how might I fix it?

Best Answer

The problem was caused by a corrupt Launch Services cache, and I solved it by executing the following command:

sudo find /System/Library/Frameworks -type f -name lsregister -exec {} -kill -seed -r \;

The clue was that the segfault was occurring at CSStoreGetUnit + 84 in both processes; a quick Google search leads to a blog entry which suggested cache corruption might be the issue. Rather than manually rming the cache files, I followed the instructions I found at The X Lab, which amounted to a detailed explanation of how to open terminal to run the aforementioned command, noting:

  • There is a smartquotes-related typo on that page (reported).
  • Since mdworker ran fine as UID 501 (and several others), I guessed I'd have to reset root's launch services cache; prefixing sudo had the desired effect.

Additional notes:

  • On 10.8.6 (at least) you can see all Launch Service cache files with the following command:

    sudo find /var/folders /Library/Caches/ -name '*LaunchServices*' -print0 |sudo xargs -0 ls -l

  • For some unknown reason, a recently-modified cache file for UID 501 exists in both /Library/Caches/ and /var/folders/; other UIDs have only one under /var/folders/. This does not seem to cause any problems.

  • This did solve the problem with backupd.

Related Question