Short answer: if your model of MacBook has a 64-bit capable CPU, you can run 64-bit apps under 10.5.8. You can check the CPU with the command
sysctl hw.cpu64bit_capable (
0 means no,
1 means yes) or by looking up the model (if it has a "Core Solo" or "Core Duo" CPU it's 32-bit only; if it's a "Core 2 Duo" -- note the "2" -- it's 64-bit).
Long answer: OS X doesn't have 32-bit and 64-bit versions; it uses a universal binary format which allows programs, system components, etc to be installed with both 32- and 64-bit code, and the system simply picks the "best" one to use based on the CPU you're on. Over different versions of OS X, what comes with the OS has gradually migrated from 32-bit-only (through 10.2) to having minimal support for background-only 64-bit apps (10.3 and 10.4), to full support for 64-bit Cocoa apps (10.5), to a full 32/64 OS including the kernel (10.6) to actually starting to drop support for 32-bit mode (10.7 shipped with many 64-bit-only system apps, and will not run on a 32-bit-only CPU).
The 64-bit-capable kernel that was added in 10.6 is cool, but basically irrelevant to user applications. OS X is unusual in that it can transparently run 64-bit apps under a 32-bit kernel as well as 32-bit apps under a 64-bit kernel. Since 10.5 has full support for 64-bit Cocoa apps, upgrading to 10.6 is not necessary.
Adendum: even if you do upgrade to 10.6, the kernel may not actually run in 64-bit mode -- Apple configured it to boot in 32-bit mode on most models by default. If your model uses 64-bit EFI firmware, you can configure 10.6 to boot in 64-bit mode, but if it's an older one that uses 32-bit EFI, even that's not possible. But again, it doesn't really matter, since 64-bit apps work fine either way.
The answer, as discovered by shizmob, is that Apple moved the location of this preference in Maverics to
/Library/Preferences/com.apple.security. So what you need to run is
sudo defaults write /Library/Preferences/com.apple.security RSAMaxKeySize -int 8192
You can use either Parallels Desktop or VMware Fusion to run older macOS versions. The virtualization runs on top of Catalina so there won‘t be a need to reboot. And unless your games are very GPU-intensive you should get decent performance as well.