Have you checked to make sure your MacBook is starting Lion up in 64-bit kernel mode? You can use this tool to check that.
Lion, even though it's a 64-bit OS, capable of running 64-bit apps, can start it's kernel in 32-bit mode on certain older machines. This won't affect most 64-bit Mac apps. They still run in 64-bit mode and can use >4GB of memory. But depending on which kernel is being used, your virtual machine will run in either a 64-bit or 32-bit process since VirtualBox interacts with the kernel at a lower level than most apps.
I have an older MacMini (2007, which boots to a 32-bit kernel) with Lion and VirtualBox on it. I've noticed that the Mac itself still performs pretty well. A 32-bit WinXP machine with 1 CPU and 1024MB RAM works almost as if it were natively on the hardware (except for graphics).
However, when I put a 64-bit Ubuntu machine onto it (even Ubuntu Server), it uses about 30% of a physical CPU core when idle and works slowly. I checked the process type in Activity Monitor, and it was a 32-bit process. The same virtual machine is very snappy on my newer iMac (booting 64-bit), and uses about 6% of a core at idle.
If you are starting up with a 32-bit kernel, your best bet might be to use 32-bit Windows in a VM. If you really need 64-bit for some reason, stick with dual booting.
Best Answer
I think that other factors, such as the hard drive arrangement you are using, will have a much greater impact on performance.
A 64-bit guest OS is unlikely to slow down the host OS any more or less than a 32-bit guest, unless you allocate more memory or CPU cores to the virtual machine. However, if you install a 64-bit guest you at least have the option for the guest OS to address more than ~4GB of memory; depending on the application (and your hardware), this could improve performance.