The problem is caused by the TasksMax systemd attribute. It was introduced in systemd 228 and makes use of the cgroups pid subsystem, which was introduced in the linux kernel 4.3. A task limit of 512 is thus enabled in systemd if kernel 4.3 or newer is running. The feature is announced here and was introduced in this pull request and the default values were set by this pull request. After upgrading my kernel to 4.3, systemctl status docker displays a Tasks line:

# systemctl status docker
● docker.service - Docker Application Container Engine
   Loaded: loaded (/etc/systemd/system/docker.service; disabled; vendor preset: disabled)
   Active: active (running) since Fri 2016-01-15 19:58:00 CET; 1min 52s ago
     Docs: https://docs.docker.com
 Main PID: 2770 (docker)
    Tasks: 502 (limit: 512)
   CGroup: /system.slice/docker.service

Setting TasksMax=infinity in the [Service] section of docker.service fixes the problem. docker.service is usually in /usr/share/systemd/system, but it can also be put/copied in /etc/systemd/system to avoid it being overridden by the package manager.

A pull request is increasing TasksMax for the docker example systemd files, and an Arch Linux bug report is trying to achieve the same for the package. There is some additional discussion going on on the Arch Linux Forum and in an Arch Linux bug report regarding lxc.

DefaultTasksMax can be used in the [Manager] section in /etc/systemd/system.conf (or /etc/systemd/user.conf for user-run services) to control the default value for TasksMax.

Systemd also applies a limit for programs run from a login-shell. These default to 4096 per user (will be increased to 12288) and are configured as UserTasksMax in the [Login] section of /etc/systemd/logind.conf.

WARNING DO NOT ATTEMPT TO RUN THIS ON A PRODUCTION MACHINE. JUST DON'T. Warning: To try any "bombs" make sure ulimit -u is in use. Read below^[a].

Let's define a function to get the PID and date (time):

bize:~$ d(){ printf '%7s %07d %s\n' "$1" "$BASHPID" "$(date +'%H:%M:%S')"; }

A simple, non-issue bomb function for the new user (protect yourself: read ^[a]):

bize:~$ bomb() { d START; echo "yes"; sleep 1; d END; } >&2

When that function is called to be executed works as this:

bize:~$ bomb
  START 0002786 23:07:34
yes
    END 0002786 23:07:35
bize:~$

The command date is executed, then a "yes" is printed, an sleep for 1 second, then the closing command date, and, finally, the function exits printing a new command prompt. Nothing fancy.

| pipe

When we call the function like this:

bize:~$ bomb | bomb
  START 0003365 23:11:34
yes
  START 0003366 23:11:34
yes
    END 0003365 23:11:35
    END 0003366 23:11:35
bize:~$

Two commands get started at some time, the two end 1 second later and then the prompt returns.

That's the reason for the pipe |, to start two processes in parallel.

& background

If we change the call adding an ending &:

bize:~$ bomb | bomb &
[1] 3380
bize:~$
  START 0003379 23:14:14
yes
  START 0003380 23:14:14
yes
    END 0003379 23:14:15
    END 0003380 23:14:15

The prompt returns immediately (all the action is sent to the background) and the two commands get executed as before. Please note the value of "job number" [1] printed before the PID of the process 3380. Later, the same number will be printed to indicate that the pipe has ended:

[1]+  Done                    bomb | bomb

That is the effect of &.

That is the reason of the &: to get processes started faster.

Simpler name

We can create a function called simply b to execute the two commands. Typed in three lines:

bize:~$ b(){
> bomb | bomb
> }

And executed as:

bize:~$ b
  START 0003563 23:21:10
yes
  START 0003564 23:21:10
yes
    END 0003564 23:21:11
    END 0003563 23:21:11

Note that we used no ; in the definition of b (the newlines were used to separate elements). However, for a definition on one line, it is usual to use ;, like this:

bize:~$ b(){ bomb | bomb ; }

Most of the spaces are also not mandatory, we can write the equivalent (but less clear):

bize:~$ b(){ bomb|bomb;}

We can also use a & to separate the } (and send the two processes to the background).

The bomb.

If we make the function bite its tail (by calling itself), we get the "fork bomb":

bize:~$ b(){ b|b;}       ### May look better as b(){ b | b ; } but does the same.

And to make it call more functions faster, send the pipe to the background.

bize:~$ b(){ b|b&}       ### Usually written as b(){ b|b& }

If we append the first call to the function after a required ; and change the name to : we get:

bize:~$ :(){ :|:&};:

Usually written as :(){ :|:& }; :

Or, written in a fun way, with some other name (a snow-man):

☃(){ ☃|☃&};☃

The ulimit (which you should have set before running this) will make the prompt return quite quickly after a lot of errors (press enter when the error list stops to get the prompt).

The reason of this being called a "fork bomb" is that the way in which the shell starts a sub-shell is by forking the running shell and then calling exec() to the forked process with the command to run.

A pipe will "fork" two new processes. Doing it to infinity causes a bomb.
Or a rabbit as was originally called because it reproduces so quickly.

Timing:

1. :(){ (:) | (:) }; time :
   Terminated
   real 0m45.627s

2. :(){ : | :; }; time :
   Terminated
   real 0m15.283s

3. :(){ : | :& }; time :
   real 0m00.002 s
   Still Running

Your examples:

1. :(){ (:) | (:) }; :

   Where the second closing ) separates the } is a more complex version of :(){ :|:;};:. Each command in a pipe is called inside a sub-shell anyway. Which is the effect of the ().

2. :(){ : | :& }; :

   Is the faster version, written to have no spaces: :(){(:)|:&};: (13 characters).

3. :(){ : | : }; : ### works in zsh but not in bash.

   Has a syntax error (in bash), a metacharacter is needed before the closing },
   as this:

    :(){ : | :; }; :
   

^[a] Create a new clean user (I'll call mine bize). Login to this new user in a console either sudo -i -u bize, or:

$ su - bize
Password: 
bize:~$

Check and then change the max user processes limit:

bize:~$ ulimit -a           ### List all limits (I show only `-u`)
max user processes              (-u) 63931
bize:~$ ulimit -u 10        ### Low
bize:~$ ulimit -a
max user processes              (-u) 1000

Using only 10 works as is only one solitary new user: bize. It makes easier to call killall -u bize and get the system rid of most (not all) bombs. Please do not ask which ones still work, I will not tell. But still: Is quite low but on the safe side, adapt to your system.
This will ensure that a "fork bomb" will not collapse your system.

Further reading:

• About forking in the bash "fork bomb"
• How does a fork bomb work?
• Why is the pipe needed?
• Understanding Bash fork() Bomb ~ :(){ :|:& };: