Spotlight, infinite indexing, secondary drives, and NAS

spotlight

I have several Mac OS X 10.5.8 editing bays, each with at least one additional internal drive, most sized 500GB-1TB. All Macs are also sharing a NAS Isilon cluster 100TB (nfs mount) for shared media etc. Seems like each workstation is running Spotlight ad-infinitum, no matter what exceptions (or none) I add into the list. On one box in particular I stopped the service in Acivity Monitor, only for it to re-spawn few secs later. I added ALL drives to exceptions list (including startup volume), Spotlight stops, but has no index of anything. I remove the startup volume, Spotlight starts and will go on indefinitely (msg says -2183764923 hours left)(15 minutes when it restarts, then just grows). SO the issue is with that volume, but what? Activity Monitor indicates 0.0 CPU for mdworker, 0.0 for Spotlight. When I kill and restart, Spotlight goes to 0.8 CPU with 18 threads and mdworker remains at 0.0 with 4 threads

Open Files and Ports yields:

/
/System/Library/CoreServices/Spotlight.app/Contents/MacOS/Spotlight
/System/Library/PrivateFrameworks/Calculate.framework/Versions/A/Calculate
/System/Library/PrivateFrameworks/URLMount.framework/Versions/A/URLMount
/System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/Resources/libCGXCoreImage.A.dylib
/System/Library/Fonts/Helvetica.dfont
/System/Library/Fonts/LucidaGrande.dfont
/System/Library/CoreServices/Spotlight.app/Contents/Resources/MDShowAll.tif
/private/var/folders/kI/kICeUX+DFmSI3L1sRRM6NE++0vw/-Caches-/com.apple.ATS/annex_aux
/private/var/folders/kI/kICeUX+DFmSI3L1sRRM6NE++0vw/-Caches-/com.apple.IntlDataCache.le.sbdl
/System/Library/Caches/com.apple.IntlDataCache.le.kbdx
/System/Library/Frameworks/Carbon.framework/Versions/A/Frameworks/HIToolbox.framework/Versions/A/Resources/HIToolbox.rsrc
/System/Library/Frameworks/Carbon.framework/Versions/A/Frameworks/HIToolbox.framework/Versions/A/Resources/English.lproj/Localized.rsrc
/Library/Dictionaries/Apple Dictionary.dictionary/Contents/Resources/English.lproj/KeyText.index
/Library/Dictionaries/Apple Dictionary.dictionary/Contents/Resources/English.lproj/KeyText.data
/System/Library/CoreServices/Spotlight.app/Contents/Resources/Index_Dot_Black.tif
/usr/share/icu/icudt36l.dat
/System/Library/Frameworks/Carbon.framework/Versions/A/Frameworks/HIToolbox.framework/Versions/A/Resources/Extras2.rsrc
/System/Library/Keyboard Layouts/AppleKeyboardLayouts.bundle/Contents/Resources/AppleKeyboardLayouts-L.dat
/Library/Dictionaries/Oxford American Writer's Thesaurus.dictionary/Contents/KeyText.index
/System/Library/PrivateFrameworks/CoreUI.framework/Versions/A/Resources/SArtFile.bin
/System/Library/CoreServices/RawCamera.bundle/Contents/MacOS/RawCamera
/System/Library/PrivateFrameworks/CoreUI.framework/Versions/A/Resources/ArtFile.bin
/Library/Dictionaries/New Oxford American Dictionary.dictionary/Contents/KeyText.index
/Library/Dictionaries/New Oxford American Dictionary.dictionary/Contents/KeyText.data
/Library/Dictionaries/New Oxford American Dictionary.dictionary/Contents/Body.data
/Library/Caches/com.apple.LaunchServices-0233007.csstore
/Library/Dictionaries/Oxford American Writer's Thesaurus.dictionary/Contents/KeyText.data
/usr/lib/dyld
/private/var/db/dyld/dyld_shared_cache_i386
/dev/null

/System/Library/Frameworks/Carbon.framework/Versions/A/Frameworks/HIToolbox.framework/Versions/A/Resources/Extras2.rsrc
/System/Library/Frameworks/Carbon.framework/Versions/A/Frameworks/HIToolbox.framework/Versions/A/Resources/HIToolbox.rsrc
/System/Library/Frameworks/Carbon.framework/Versions/A/Frameworks/HIToolbox.framework/Versions/A/Resources/English.lproj/Localized.rsrc

Sampling yields:

Sampling process 74486 for 3 seconds with 1 millisecond of run time between samples
Sampling completed, processing symbols...
Analysis of sampling Spotlight (pid 74486) every 1 millisecond
Call graph:
    1590 Thread_2507
      1590 0x26ea
        1590 NSApplicationMain
          1590 -[NSApplication run]
            1590 -[NSApplication nextEventMatchingMask:untilDate:inMode:dequeue:]
              1590 _DPSNextEvent
                1590 BlockUntilNextEventMatchingListInMode
                  1590 ReceiveNextEventCommon
                    1590 RunCurrentEventLoopInMode
                      1590 CFRunLoopRunInMode
                        1590 CFRunLoopRunSpecific
                          1570 mach_msg
                            1570 mach_msg_trap
                              1570 mach_msg_trap
                          16 __CFRunLoopDoObservers
                            16 _handleWindowNeedsDisplay
                              16 -[NSWindow displayIfNeeded]
                                16 -[NSView displayIfNeeded]
                                  14 -[NSView _displayRectIgnoringOpacity:isVisibleRect:rectIsVisibleRectForView:]
                                    10 -[NSNextStepFrame _recursiveDisplayRectIfNeededIgnoringOpacity:isVisibleRect:rectIsVisibleRectForView:topView:]
                                      9 -[NSView _recursiveDisplayRectIfNeededIgnoringOpacity:isVisibleRect:rectIsVisibleRectForView:topView:]
                                        9 -[NSView _recursiveDisplayAllDirtyWithLockFocus:visRect:]
                                          7 -[NSView _drawRect:clip:]
                                            7 NSRectFill
                                              6 CGContextFillRect
                                                6 CGContextFillRects
                                                  6 ripc_DrawRects
                                                    6 ripc_Render
                                                      6 ripl_BltShape
                                                        6 ripd_Lock
                                                          6 CGSDeviceLock
                                                            6 _CGSLockWindow
                                                              6 _CGSSynchronizeWindowBackingStore
                                                                6 mach_msg
                                                                  6 mach_msg_trap
                                                                    6 mach_msg_trap
                                              1 CGContextGetCompositeOperation
                                                1 CGContextGetCompositeOperation
                                          2 -[NSView _recursiveDisplayAllDirtyWithLockFocus:visRect:]
                                            2 -[NSView _drawRect:clip:]
                                              1 CGContextRestoreGState
                                                1 CGGStackRestore
                                                  1 CGGStateRelease
                                                    1 _CFRelease
                                                      1 _CFRelease
                                              1 NSRectClipList
                                                1 CGContextClipToRects
                                                  1 CGContextClipToRect
                                                    1 CGGStateClipToRect
                                                      1 maybeCopyClipState
                                                        1 CGClipStackCreateMutableCopy
                                                          1 CGClipStackCreateMutableCopy
                                      1 objc_msgSend
                                        1 objc_msgSend
                                    2 -[NSWindow flushWindow]
                                      2 CGContextFlush
                                        2 ripc_Operation
                                          2 CGSDeviceSynchronize
                                            2 _CGSFlushRegionInline
                                              2 mach_msg
                                                1 mach_msg
                                                1 mach_msg_trap
                                                  1 mach_msg_trap
                                    1 -[NSRectSet initWithRegion:]
                                      1 -[NSRegion getRects:count:]
                                        1 CGSRegionEnumerator
                                          1 CGSRegionEnumeratorWithDirection
                                            1 CGSRegionEnumeratorCreate
                                              1 CGTypeCreateInstanceWithAllocator
                                                1 _CFRuntimeCreateInstance
                                                  1 _CFRuntimeCreateInstance
                                    1 -[NSView lockFocus]
                                      1 -[NSView lockFocusIfCanDraw]
                                        1 PSgsave
                                          1 CGContextSaveGState
                                            1 CGGStackSave
                                              1 CGGStateCreateCopy
                                                1 calloc
                                                  1 malloc_zone_calloc
                                                    1 szone_calloc
                                                      1 __bzero
                                                        1 __bzero
                                  1 objc_msgSend
                                    1 objc_msgSend
                                  1 objc_msgSend_stret
                                    1 objc_msgSend_stret
                          3 __CFRunLoopModeFindSourceForMachPort
                            3 CFSetApplyFunction
                              2 __CFRunLoopFindSource
                                2 __CFRunLoopFindSource
                              1 CFSetApplyFunction
                          1 __NSFireTimer
                            1 objc_msgSend_fpret
                              1 objc_msgSend_fpret
    1590 Thread_2603
      1590 thread_start
        1590 _pthread_start
          1590 fe_fragment_thread
            1590 pthread_cond_wait$UNIX2003
              1590 __semwait_signal
                1590 __semwait_signal
    1590 Thread_2703
      1590 thread_start
        1590 _pthread_start
          1590 fe_fragment_thread
            1590 pthread_cond_wait$UNIX2003
              1590 __semwait_signal
                1590 __semwait_signal
    1590 Thread_2803
      1590 thread_start
        1590 _pthread_start
          1590 fe_fragment_thread
            1590 pthread_cond_wait$UNIX2003
              1590 __semwait_signal
                1590 __semwait_signal
    1590 Thread_2903
      1590 thread_start
        1590 _pthread_start
          1590 fe_fragment_thread
            1590 pthread_cond_wait$UNIX2003
              1590 __semwait_signal
                1590 __semwait_signal
    1590 Thread_2a03
      1590 thread_start
        1590 _pthread_start
          1590 fe_fragment_thread
            1590 pthread_cond_wait$UNIX2003
              1590 __semwait_signal
                1590 __semwait_signal
    1590 Thread_2b03
      1590 thread_start
        1590 _pthread_start
          1590 fe_fragment_thread
            1590 pthread_cond_wait$UNIX2003
              1590 __semwait_signal
                1590 __semwait_signal
    1590 Thread_2c03
      1590 thread_start
        1590 _pthread_start
          1590 fe_fragment_thread
            1590 pthread_cond_wait$UNIX2003
              1590 __semwait_signal
                1590 __semwait_signal
    1590 Thread_2d03
      1590 start_wqthread
        1590 __workq_ops
          1590 __workq_ops
    1590 Thread_2e03
      1590 start_wqthread
        1590 __workq_ops
          1590 __workq_ops
    1590 Thread_2f03
      1590 start_wqthread
        1590 __workq_ops
          1590 __workq_ops
    1590 Thread_3003
      1590 0x0
        1590 0x0
    1590 Thread_3103
      1590 0x0
        1590 0x0
    1590 Thread_3203
      1590 0x0
        1590 0x0
    1590 Thread_3303
      1590 0x0
        1590 0x0
    1590 Thread_3403
      1590 0x0
        1590 0x0
    1590 Thread_3503
      1590 thread_start
        1590 _pthread_start
          1590 __NSThread__main__
            1590 -[NSThread main]
              1590 0x336c7
                1590 CFRunLoopRunInMode
                  1590 CFRunLoopRunSpecific
                    1590 mach_msg
                      1590 mach_msg_trap
                        1590 mach_msg_trap
    1590 Thread_3603
      1590 thread_start
        1590 _pthread_start
          1590 __NSThread__main__
            1590 -[NSThread main]
              1590 -[NSUIHeartBeat _heartBeatThread:]
                1590 -[NSConditionLock lockWhenCondition:]
                  1590 -[NSConditionLock lockWhenCondition:beforeDate:]
                    1590 -[NSCondition waitUntilDate:]
                      1590 pthread_cond_timedwait_relative_np
                        1590 _pthread_cond_wait
                          1590 semaphore_timedwait_signal_trap
                            1590 semaphore_timedwait_signal_trap

Total number in stack (recursive counted multiple, when >=5):

9       _pthread_start
9       thread_start
7       __semwait_signal
7       fe_fragment_thread
7       pthread_cond_wait$UNIX2003
5       0x0

Sort by top of stack, same collapsed (when >= 5):

__semwait_signal        11130
0x0                     7950
__workq_ops             4770
mach_msg_trap           3167
semaphore_timedwait_signal_trap        1590

Sample analysis of process 74486 written to file /dev/stdout

===========================================================================

I do want to index all drives for content. I would also like a method of indexing the NAS, as this would be particularly useful. Anyone has any suggestions on how to implement Spotlight to actually do its job locally, and even possibly on removable FW drives, or even a NAS?

Best Answer

There a command line tool which you could use: mdutil. Open Terminal.app application and try the following:

To check the indexing status (on|off) for a volume:

thomas$ mdutil -s /Volumes/nfs-spotlight/
/Volumes/nfs-spotlight:
    Indexing enabled.

...and to turn indexing off:

thomas$ sudo mdutil -i off /Volumes/nfs-spotlight/
Password:
/Volumes/nfs-spotlight:
    Indexing disabled.

...or even better disable spotlight activities:

thomas$ sudo mdutil -d /Volumes/nfs-spotlight/
/Volumes/nfs-spotlight:
    Indexing and searching disabled.

Calling mdutil without any parameter shows all the allowed parameters

It works with my NAS nfs mount but then it's much smaller than 100TB. It's worth a try nevertheless.