When a computer goes into hibernation mode, it saves the contents of the RAM into the swap space so that it can resume exactly where it left off when it powers back on.
So, what if you are currently using more memory than you have RAM. In this case, some data that would otherwise be in RAM is saved to swap.
An example. A computer has 1GB of RAM and 1GB of swap space. It is currently using all but 100MB of RAM and 500MB of swap. Thus, it is needing to remember about 1.4GB of memory to save its state. However, swap is only 1GB.
What will happen in this case?
~~ edit ~~
As answered below, the hibernate fails. So then a follow-up question:
When this fails, where is the error reported?
Best Answer
That depends on the implementation of hibernation. Even if you restrict the question to Linux, the implementation has evolved over time.
First, consider that some of the RAM is used for disk caches. This doesn't need to be moved to the swap as it can be reloaded from the disk after the system resumes. On a system with a good cost/efficiency balance, it's typical for about half of RAM being allocated to caches. (See also Why use swap when there is more than enough RAM.) Under Linux, some early implementations would store all allocated memory into the swap, but the current implementation(s?) of hibernation skip disk caches.
Second, some systems compress memory as it's written to the swap, which can make the exact required amount of swap hard to predict. Some versions of Linux's hibernation support have supported compression; I don't know if current ones do.
What you can generally expect if there isn't enough swap space is that hibernation will fail: the system will try to store the (useful) contents of RAM into the swap, and as soon as it detects that there isn't enough space, the hibernation attempt is aborted (typically with an error message on the console and in the system logs). As far as I know, Linux has always behaved like this (not that there's really another sensible behavior).