Is it necessary to reboot after a kernel upgrade (via APT)

You forgot to build your initrd that goes with the kernel. Run update-initramfs -c -k kernelversion and then update-grub to find it and add it to the grub menu.

First, I’ll start by explaining the behaviour you’re seeing. By default, apt-get is very conservative and won’t install new packages when running upgrade, only dist-upgrade (this can be changed with the --with-new-pkgs option). apt on the other hand will (it enables the APT::Get::Upgrade-Allow-New option by default), as will aptitude (which has a different resolution algorithm anyway). Since the kernel went through an ABI bump, upgrading it involves installing a new package...

Note too that strictly speaking you’re not getting this kernel update as a security fix, but as part of last weekend’s 9.5 point release. That’s neither here nor there though since the priorities are the same. (You’re aware of this already, I mention this for other readers.) Security updates to the kernel can involve ABI bumps, so this situation can arise with security updates as well as point-release updates; the distinction, as far as the kernel packages in Debian are concerned, is mostly one of opportunity and timing, since all kernel updates are security updates anyway.

Now to answer your questions:

1. apt is now the recommended APT front-end, yes, at least for interactive use; it’s supposed to have user-friendlier defaults (compared with apt-get). Both tools use the same algorithms and apt-get can be configured to behave like apt; run apt-config dump | grep Binary::apt to see the specific settings apt enables. If you prefer aptitude you can keep on using that too.

2. In Debian 8, as far as I can tell, apt is exactly equivalent to apt-get, so you can use it too; apt came to the forefront with Debian 9 following work by its developers to make it more useful as a user-facing tool. The Debian FAQ suggests using apt instead of apt-get and apt-cache starting with Debian 8 (Jessie). (I see it gets the description of apt update slightly wrong.)

3. The current recommendation is to avoid apt in scripts since its interface may change. Apart from that, there shouldn’t be any scenarios where you’d need apt-get rather than apt.

4. apt doesn’t distinguish between security and non-security updates, unless you configure it to do so.

5. I’m not sure it’s a conscious decision... The last update to the Securing Debian Manual is quite a bit older than Debian 9. Note that, when using apt-get, the Debian FAQ refers to apt-get dist-upgrade to keep a Debian system up-to-date. The FAQ is also older than Debian 9, but was updated more recently.