Because OSX manages memory (as does Windows) it is often difficult to tell if performance issues are caused by memory limits. Technically, you really want the OS to capture all the available memory and allocate as needed, in which case, you would see all of the memory being 'used', even though there may be memory available but not allocated by the OS.
This is somewhat analogous to what is happening in OSX and Win7+. However, Apple has made it confusing in the past with many memory classifications, like Wired, Swap, Real, etc.
Luckily, you are running Mavericks, and a new feature of Mavericks is the Memory Pressure graph, described in the Apple support document here: http://support.apple.com/kb/HT5890
Basically, if the memory graph shows Red, you need more RAM, if its Green, you are good. I suggest reading the article for more details, but the Red, Yellow, Green is a great visual tool to help guide where you might be seeing problems.
Depending on your model, I have personally found Macs suffer more from hard drive access speed than memory limits, so an SSD, if your iMac does not have one, has a huge impact (and unfortunately quite a bit of work to put into your iMac)
The kernel_task process in question is likely your process running.
You can try freeing up "inactive memory" with the purge command in Terminal, just type purge and hit return when the inactive memory gets to be a good percentage of the used memory.
You can also try running your process without the gui, or without the window server running. This will free up memory and processor resources.
Set your login window to require name and password from the accounts control panel in system preferences, turn off auto login if enabled.
Then reboot to free up memory.
from the login window (if set as username/password) type >console in the name field, hit enter, and when the window server quits...
log in at the console, shortname & enter password when prompted
type screen and hit return
screen man page - so you can have another term to watch with top; will use less resources than running gui. control-a then c to create new terminal, control-a then n to move between the two scrreens
run your process in one, run top -u -s5 to run a utility like activity monitor so you can monitor your process
when done, just exit out of both screens, and then exit out of the console to launch the window server and return the login window and back to gui
Best Answer
I have been working with a senior technical advisor at Apple on this issue for over a year, and was working with another senior advisor for some time before that. We have done "data capture" to send to Apple Engineers on several occasions and done screen recordings on several occasions to demonstrate what is going on in Activity Monitor, Image Capture, and, ultimately, in a plist that icdd maintains at /Users/user_name/Library/Application Support/icdd/deviceInfoCache.plist (by displaying it in Xcode).
At this point, here is my best estimate of what is happening:
The icdd (Image Capture Device Database) process sees scanners come and go on a busy network. It attempts to keep a list of their icon files in a hash table, which it also writes to the deviceInfoCache.plist file mentioned above. Yes - this sounds crazy - it is keeping references to the scanners' icon files. But even crazier is that, for some reason, almost all of the entries in this file point to .icns files that do not exist. Of several systems I've looked at, there have been many thousands of entries in the file, yet only a few of the .icns files existed on one of the machines, and none existed on the others. I believe that when this file gets large, icdd is spending a lot of time trying to check for the existence of entries in the .plist file and modify the file. I believe this for two reasons. First, when I take my laptop home, the icdd process sometimes continues to run at about 100% of a CPU, but when I then kill it, it goes back to the "normal" approximately 0.0 to 0.1%, every time. Hence, I think it is sometimes still trying to process information about the entries when I open it up at home. But when I kill it while on the busy network, it often comes back at near 100% right away. When the number of scanners shown in Image Capture goes down (which it often does, but will periodically spike for some reason), icdd will eventually settle down. And second, deleting the deviceInfoCache.plist file causes icdd to behave reasonably for a short while - until the number of entries builds up again. Note that icdd maintains a copy of these entries in memory, so if you delete the file from the user account, icdd just rewrites it immediately. And, of course, you can't kill icdd long enough to delete the file, so you have to log out and delete the file from another administrator account via the terminal. icdd will recreate the file when you log back in, but it will have relatively few entries and behave well for a while.
To give some idea of scales, Apple Engineers were shocked to see that I had as many as 85 scanners displaying in Image Capture. Often, however, this number will settle down to about 6 on the same system and during the same timeframes. The deviceInfoCache.plist file has had between 8,000 and 12,600 entries on the systems that I've looked at that have had icdd problems - mine is the larger one, and I believe this got carried over from an older machine since I was having icdd issues from the time I set up my new MacBook Pro in 2016-Dec. When I deleted the plist file, the number of initial entries in the newly created file was 44, and for a few days icdd cpu usage hovered close to 0.0%. However, after about 5 days on campus, my plist file has 964 entires, and icdd cpu usage will routinely bounce between 30% and 90% on the busy network at the university. When I am at home, the plist file will only increase its number of entries by 0 to 2 over the course of a day. Of the 12,600 entries in my previous plist file, only 2 of them contain a "deviceName", the rest contain an "iconPathLocation", all of which point to nonexistent .icns files. With the current plist, there are still exactly 2 entries that contain a "deviceName", and the rest contain an "iconPathLocation" that does not exist.
So, the short-term solution is to delete the plist file from another administrator account via terminal while logged out from your user account. Hopefully, with this information now being provided to Apple Engineers from my Senior Advisor, Apple Engineers will have enough information to figure out why icdd is acting this way and fix the problem. Of course, it would probably help if you can verify my short-term solution and continue reporting what you find to Apple.