A similar question to this is noted here How Can I Limit the Amount of RAM a Mac OS Application Can Use?
They recommend using the ulimit command, but they were unable to successfully limit the amount to what they desired. I think the problem that they ran into was the fact that the option being set -u or -v was not accounting for the memory paging system, eg you can only allocate X amount of space for you process in real RAM, but we can page you out to virtual memory all day long.
Note that this may be Dangerous disabling Virtual Memory, it could result in your Mac just locking up, etc when it runs of of RAM.
Try running ulimit with arguments -l, -m, or -v however as noted in the ulimit man page it up to the OS if it will pay attention to these settings or not as noted.
ulimit: ulimit [-SHacdefilmnpqrstuvx] [limit]
Modify shell resource limits.
Provides control over the resources available to the shell and processes
it creates, on systems that allow such control.
Can't really find any definite source that confirms if OS X ignores ulimit or not, but after the referenced ulimit fail
If you can get ulimit to work for your needs by itself then give it a go, if not a combination of ulimit and or disabling virtual memory may be able to let you set up that 3 GB limit on select processes. FYI here is a good link for more information onCompletely Disable Virtual Memory in Snow Leopard System 10.6
sudo launchctl unload -w /System/Library/LaunchDaemons/com.apple.dynamic_pager.plist
The limitation imposed is based on the chipsets used on the computer's motherboard. For example, the noted Apple maximums you mentioned for the MacBook Pro and the Mac Pro, are smaller than the maximums that those machines can address in practice. The MacBook Pro tested maximum is 16 GB according to EveryMac's MacBook Pro max RAM listing. And as for the Mac Pro, its maximum is 128 GB according to EveryMac's Mac Pro max RAM listing.
As noted on Intel's Specs, the CPU supports a different maximum than the chipset on the motherboard does. The Xeon X5670 used in the Mac Pro supports up to 288 GB.
And as for the i7s e.g. a i7-2860QM used in the MacBook Pros, it supports up to 32 GB.
So these limitations come from engineering decisions made by Apple based on what kinds of chipsets are selected to be installed on the motherboard and what those chips support is what enforces the maximum amount of RAM that a particular Mac can address at or below what the CPU can actually address.
As for why Apple underrates their maximum numbers for RAM on some Macs, that's for Apple to know and for us to wonder. However, admittedly it is a nice practice to under promise and over deliver for whatever reason Apple has. Although, the Mac Pro supports more RAM than Mac OS X can address, according to OWC's testing on Mac Pros, where they discovered Mac OS X unofficially will not address more than 96GB of RAM, but other 64-bit operating system can get to the full 128GB.
Best Answer
According to Mactracker, your MacBook 13-inch, Early 2009 (A1181, early 2009) can handle 6 GB RAM, or 2 GB more than Apple's 4 GB spec. You'd need to use a combination of a 4 GB and 2 GB 200-pin PC2-5300 (667MHz) DDR2 SO-DIMM.
See also: Apple MacBook "Core 2 Duo" 2.0 13" (White-09) Specs at EveryMac.com