It sounds like some sort of RAM issue for sure, and there are a few memtest based programs out there to try and run and see if you run into problems.
Rember is a gui based memtest, and applejack is a fairly well known utility script that also contains it.
I dont know why you only got the error with a specific configuration of ram, but with 8GB of addressable space it can take a while to find the bad guy.
The obvious thing to try is eject the Time Machine volume (or otherwise make it unavailable - turned off) and reboot the mac. mds will only use RAM when a program is asking it to supply spotlight answers. The files you have are using a large amount of space and being called upon. A clean boot should get your mds with little or no RSS until you start to launch apps or TM starts.
Whether this is normal is harder to tell - even if it's caused by your files, it's likely made worse a direct result of the PPC architecture not having as optimized a time of indexing than the Intel architecture. The mds program likely is coded with parts of the code for PPC and part for Intel as part of the normal performance optimizations Apple or any large corporation will do before releasing a daemon that runs all the time. Furthermore, the frameworks it uses will be native to the processor - so it's not really the same program running when you have a different CPU.
The comments to the question show that you have really down your homework and played around a bit with excluding volumes to see if the large memory is happening in response to many drives being indexed.
Sadly, you don't have any real control over how large the process gets. It simply runs based on the spotlight importers that are present on your system and in response to all the filesystems you expose the spotlight subsystem to.
If the obvious step of isolating Time Machine doesn't reduce the RAM usage, You can also try:
- cleaning out third party spotlight extensions
- add more RAM and live with it
- permanently disable the folders that are causing RAM usage to spike (and forego the usefulness of spotlight)
- temporarily disabling all volumes by putting them on the privacy exclusion list and then slowly adding folders to be indexed and hope to determine which sort of data is causing the worst spike in
- use Shark and their kind (part of Xcode and the CHUD tools) to sample the running mds process and map out the memory allocation as it is happening. This is fairly high on the skill level and won't change your end solution either way - you'll just know why - and maybe still not have a cure.
Also sadly for you - most people are not running 10.5 on PPC chips, so the number of people that can run some tests or have seen this and poked around the issue it to find out what sort of file or importer might be likely to take more memory once mds gets it's hands on the metadata store.
It does sound odd - so you have a very valid point with more than usual RAM usage. But, it's not necessarily something that is wrong - you just might have data that needs that large amount of RAM. Only by taking your data to a similar mac or doing the process of elimination can you know for sure.
Best Answer
I've been a fan of Apple's Hardware Test (AHT), built into newer Macs and available on restore DVDs for older Macs, for several years now. It allows for both a quick test (basic memory tests, basic test for other components and a quick HD scan) and an extended test. The extended test runs full verification tests on the RAM, the basic hardware tests and a full disk scan of the hard drive.
Apple has a knowledge base article devoted to AHT: Using Apple Hardware Test -HT1509
Basically depending on the age of your Mac you have two options...
Older Macs that shipped with a gray restore disc/s or MacBook Airs that shipped with a USB recovery disk:
Insert the gray restore DVD (or USB recovery disk if MacBook Air)
Power on your Mac while holding D
It may take several (15+) seconds for AHT to load, once loading though you should see this icon appear on the screen. If you see the Apple logo then you will need to shutdown the Mac and try again.
Select your desired language
There are three tabs. About explains the tool. Hardware tests runs the tests. Hardware Profile shows the Macs configuration. The middle tab, Hardware tests, is the one you'll use
You have two options, quick test (the default setting) or a checkbox to run an extended test. The extended test takes considerably longer (typically an hour although this will vary based on hard drive size and amount of installed RAM) but is more thorough.
If the tests pass you will receive a "No trouble found" message. If the tests fail you will receive a somewhat cryptic error code. This Cnet article does a great job of explaining the error codes. For what you're interested in though you'd be looking for a failure beginning with 4/MEM, this would indicate a failure with the memory test, either with the DIMM or the DIMM slot on the logic board.
Newer Macs that shipped with 10.7 or above, or without restore media, will follow the same steps as above however they can ignore the very first step since there is no media to insert in the Mac.
Newer, post June 2013, Macs will instead use Apple Diagnostics. While similar to AHT in it's capabilities it lacks the option for an extended test. You invoke it in the same manner as AHT.
If you're instead looking for a tool to run from within your OS then my tool of choice has always been TechTool Pro. It's memory tests usually confirm suspected RAM issues.
Hopefully that helps!