After some grepping I found the solution: You need to add the numpad:microsoft option to the XkbOptions. On older Ubuntus, do that in your xorg.conf. On newer ones open the file /etc/default/keyboard and change this line:
Save and reboot (restarting X doesn't seem to work, at least not with RAlt+PrintScreen+K). You may need to run sudo dpkg-reconfigure keyboard-configuration for changes to take effect.