How to Prevent a Fork Bomb

Justin's answer tells you how to raise the number of open files available total to the whole system. But I think you're asking how to raise the per-user limit, globally. The answer to that is to add the following lines to /etc/security/limits.conf:

*               soft    nofile            2048
*               hard    nofile            2048

(Where the * means all users.)

There's some summary documentation in that file itself and in man limits.conf. This is implemented via the pam_limits.so module which is called for various services configured in /etc/pam.d/.

And, I have to admit, I have no idea where that 1024 default comes from. And believe me, I looked. I even tried without the pam_limits module configured, and it's still there. It must be hard-coded in somewhere, but I'm not exactly sure where.

Is there anyway to stop this without rebooting the machine?

It's not quite impossible, and you can do it via luck -- i.e., you manage to kill all the processes before another one is spawned.¹ But you have to get very very lucky, so it is not a reliable or worthwhile effort [maybe slm is luckier than me here, lol -- TBH I haven't tried that hard]. If you play around with priorities, your chances could improve (see man nice), although I suspect this will also mess with the efficacy of the fork bomb.

A better idea might be to use one that times out. For an example in C, see footnote number 5 to my answer here.² You can do the same thing with a shell script, albeit would not be as short as :(){ :|:& };::

#!/bin/bash

export fbomb_duration=$1
export fbomb_start=$(date +%s)

go () {
    now=$(date +%s)
    if [[ $(($now-$fbomb_start)) -gt $fbomb_duration ]]
        then exit 0;
    fi
    go &
}

while ((1)); do
    go
done           

Execute that with one argument, a number of seconds. All forks will die after that time.

¹ In fact, it can happen all on its own, eventually, if the kernel OOM killer gets lucky. But don't hold your breath.

² The method used there to hamstring that particular bomb (by setting vm.overcommit_memory=2) will almost certainly not work in general, but you could try. I'm not since I'd like to leave my system running for now ;)