Ssh – Allow SSH access but restrict root’s access to a given set of IPs

I want to allow SSH for all users as normal except that I want to restrict root's access based on a set of IP addresses. So, users peter, paul, lary can login from anywhere, but root may login only from hosts a.b.c.d and q.r.s.t.

• sshd_config's AllowUsers won't work for this. If I specify anything, then I must specify all users.
• Unlike how the man pages suggest, AllowGroups is checked unconditionally, even if the user is whitelisted in AllowUsers; so if I tried to white list non-root users by putting them in a group, and then add the group to AllowGroups, the authentication will still fail because root is not in a valid allowed group.
• sshd_config's DenyUsers might work if I can somehow whitelist the set of IPs root is otherwise denied from. If I had only one IP, it might work with the ! operator.
• I can do this partially with the options key in the authenticated_keys file and by completely disabling the root password. The problem is that this file is not system-policy and may be overwritten by another (root-access) user. Currently, it's the best option I got, and I don't like it. Also, if I nuke the root password, someone in my group will be very angry with me. (If I don't nuke the root password, someone can login via root with the password from any IP.)

• I tried to do this with PAM, specifically via pam_listfile, but my approach didn't seem to work at all:

  auth       required     pam_sepermit.so
  auth       required     pam_listfile.so file=/etc/root-whitelist.txt sense=allow item=rhost apply=root
  auth       include      password-auth
  

  Inside the root-whitelist.txt file was allowed IP addresses, line by line. I could not get the rule to deny non-listed IPs access.

Is the pam_listfile approach usable and I simply got it wrong? Is there a better way?