Simple answer: It either will only see the max supported memory or it will not work at all.
My gut feeling says two things:
If it does work, you just will not see the extra memory, only the max the motherboard/chipset supports. If it doesn't work, it usually will just beep at you with a memory error and you will get no video responce. All depends on the motherboard depending on how it handles memory errors.
Someone on Yahoo answers says an interesting bit about getting a blue screen due to this:
In my experience, putting in more than
the max amount of RAM as specified by
the manufacturer will cause the
computer to not boot up.
Some won't POST, some will and simply
BSOD (kernel panic, etc) with
That said, the "max" memory isn't
always the actual max. Case in point,
Intel states the GL960 chipset (such
as in my laptop) supports a max of 2GB
of memory. 4GB is a no-go, but 3GB
You're likely to fry the memory chips as well as the motherboard.
If the system isn't hot-swappable, there's likely some electric interaction between the RAM and the motherboard. Removing the RAM while the current is flowing will be very likely to create sparks and strong currents that will wreak havoc on your system.
If you're lucky enough not to cause any physical damage, the system will likely hang. During power-up the BIOS determines how much physical memory is available and sets up the system to use the resources accordingly. The memory is likely to be allocated to some piece of software, either BIOS, OS or some user software. When you remove its memory, that software will lose some of its code or data, which is not healthy either.