I see only three potential benefits:
64-bit applications can grant you a performance boost, when higher numerical precision is needed. If you are only using your PC for stuff like Firefox or iTunes, you probably won't get a boost but if you are using some scientific or higher-demanding applications, it is definitely the way to go.
Drivers are usually more stable since Microsoft requires 64-bit drivers to be certificated which does mean that at least some stress testing was done. 32-bit Windows do not have that requirement so manufacturers tend to be cheap. Unfortunately this also means that new drivers are usually first available for 32-bit version and 64-bit version may come little bit later.
If you are developer, it may be useful for testing your application in 64-bit environment. When you have 64-bit Windows, you can test both 32-bit and 64-bit behavior.
Personally, I would stick with 32-bit Windows in this particular case unless you have very good reason.
What I understand about 32-bit OS is, the address is expressed in 32 bits, so at most the OS could use 2^32 = 4GB memory space
The most that the process can address is 4GB. You are potentially confusing memory with address space. A process can have more memory than address space. That is perfectly legal and quite common in video processing and other memory intensive applications. A process can be allocated dozens of GB of memory and swap it into and out of the address space at will. Only 2 GB can go into the user address space at a time.
If you have a four-car garage at your house, you can still own fifty cars. You just can't keep them all in your garage. You have to have auxiliary storage somewhere else to store at least 46 of them; which cars you keep in your garage and which ones you keep in the parking lot down the street is up to you.
Does this mean any 32-bit OS, be it Windows or unix, if the machine has RAM + page file on hard disk more than 4GB, for example 8GB RAM and 20GB page file, there will never be "memory used up"?
Absolutely it does not mean that. A single process could use more memory than that! Again the amount of memory a process uses is almost completely unrelated to the amount of virtual address space a process uses. Just like the number of cars you keep in your garage is completely unrelated to the number of cars you own.
Moreover, two processes can share non-private memory pages. If twenty processes all load the same DLL, the processes all share the memory pages for that code. They don't share virtual memory address space, they share memory.
My point, in case it is not clear, is that you should stop thinking of memory and address space as the same thing, because they're not the same thing at all.
if this 32-bit OS machine has 2GB RAM and 2GB page file, increasing the page file size won't help the performance. Is this true?
You have fifty cars and a four-car garage, and a 100 car parking lot down the street. You increase the size of the parking lot to 200 spots. Do any of your cars get faster as a result of you now having 150 extra parking spaces instead of 50 extra parking spaces?
I don't recommend using it, it's not stable. But if you do, good luck.
SuperUser Post Covering the Same topic (PAE)