I suspect you probably are using Mac OS 10.7 which is very memory hungry. As noted by kccqzy, the purge command may reclaim some inactive memory if you have XCode installed.
Something you should try is closing and reopening applications which are memory hungry or have a tendency to leak memory (Safari, Firefox). Alternatively, restarting your computer would also accomplish the same though take considerably longer.
To answer the question, are you really using 4GB of memory. Yes. Technically more if you take note of the "Swap used:" stat.
I've been researching this very question for some time, and I've come to the conclusion that while there are several so-called "solutions" offered in various places, none of them really fixes the problem or makes the symptoms go away. The best I've found is techniques that change when inactive memory is swapped, such as the use of the purge command you referred to. I believe that MacLemon is correct that swapping can be disabled only in entirety and not selectively, and so disabling the swapping of inactive memory would in practice be equivalent to disabling virtual memory altogether—which could be a workable solution on a machine with really large amounts of RAM, but is impractical on machines with a low maximum RAM capacity, such as MacBooks or Minis.
The reason that none of the ad-hoc purging solutions make the situation significantly better is that there are really only two ways to force OS X to clear inactive memory: the purge command, or forcing allocation of all free memory (and since I'm not certain what method purge uses, these two may actually be more or less the same). purge, as you mentioned, takes a non-trivial amount of time to complete. Allocating all free memory simply accelerates the process by which the contents of inactive memory would be swapped out naturally, and so still takes up the same large amount of system resources. The only advantage of either of these solutions is that they allow you to choose when the swapping occurs, so it can be done before you would be normally trying to allocate that memory to something else. Whether that is actually useful to you depends on several factors, so the simplest way to find out is to just try it.
I've been testing a few of the memory-clearing utilities out there, and I've found that for me, manually forcing inactive swap requires more active monitoring of memory levels than is practical while I'm actually working, and using a utility that automatically forces the swap when free memory drops below a certain threshold is no better than letting the OS do it on its own, as I still have no control over when the swapping will occur and my apps will SPOD. So while there is an app that will do exactly what your alternative question asks for, that doesn't actually make the situation any less painful.
Until Apple's OS development team decides that the memory management system isn't working the way it's supposed to, and they figure out a way to make it work properly, the only real solution is to identify which apps are generating the most inactive memory and stop using them. In my situation, this has meant changing web browsers. I've been testing a variety of them, and Chrome is so far the one that seems to generate the least inactive memory, probably in part because every tab and every extension runs as a separate process, allowing the OS's native memory management to treat each one separately in terms of prioritizing swapping. Safari is the worst I've tried; starting with Safari 5.1, I could open a few tabs, do absolutely nothing, and watch the inactive memory rise rapidly in Activity Monitor—it would easily go from <1gb inactive to >3gb inactive within about five minutes, run the swap cycle, and then do the exact same thing. (Granted, memory that is doing nothing is exactly what makes it inactive in the first place, but it shouldn't create more inactive memory than was allocated active in the first place.) Safari 6 under Mountain Lion is a bit better, but not better enough to be worth switching back. Firefox, not being based on WebKit, ought to be better about this than either Safari or Chrome, but it has its own legacy of memory management problems, including a history of memory leaks, that make it no better in practice.
What would really fix the issue is if there was an option, likely a hidden option in the OS, telling the OS to simply dump the contents of inactive memory when needed instead of swapping its contents to disk. But I don't expect Apple to ever make such an option available.
Best Answer
In general, operating systems tend to work towards having "cache + memory used" equal the actual amount of physical memory. The more that is cached, the greater the likelihood that certain operations can be sped up.
In general having more RAM means that the operating system with the same work load is able to cache more things, which could potentially speed some operations up. It is not however an indicator that you "need" more RAM as such.
However, your screenshots does indicate that you're using 1.8 GB of swap. That could be an indicator that more RAM is necessary. Coupled with the fact that you experience a lag when switching apps - it does sound likely that more RAM could be helpful.
I would try using the Terminal and running the command
top. It will show a list of processes running on your Mac, and at the top it will list the number of "swapins" and "swapouts". If you have top running in a visible window, and then try switching between the apps that you use - then check if the those two numbers start rising when you experience the lag. If that's the case, I think a RAM upgrade would most probably improve the situation.