Although POSIX has a standard for capabilities which I think includes CAP_NET_BIND_SERVICE, these are not required for conformance and may in some ways be incompatible with the implementation on, e.g., linux.
Since webservers like apache are not written for only one platform, using root privileges is the most portable method. I suppose it could do this specifically on linux and BSD (or wherever support is detected), but this would mean the behaviour would vary from platform to platform, etc.
It seems to me you could configure your system so that any web server could be used this way; there are some (perhaps clumsy) suggestions about this WRT apache here: NonRootPortBinding.
So why are they traditionally being started as root when afterwards everything is done to get rid of implied security issues that come with it?
They're started as root because they usually need to access a privileged port, and traditionally this was the only way to do it. The reason they downgrade afterward is because they do not need privileges subsequently, and to limit the damage potential introduced by the myriad of third party add-on software commonly used by the server.
This is not unreasonable, since the privileged activity is very limited, and by convention many other system daemons run root continuously, including other inet daemons (e.g., sshd).
Keep in mind that if the server were packaged so that it could be run as an unprivileged user with CAP_NET_BIND_SERVICE, this would allow any non-privileged user to start HTTP(S) service, which is perhaps a greater risk.
Root is user 0
The key thing is the user ID 0. There are many places in the kernel that check the user ID of the calling process and grant permission to do something only if the user ID is 0.
The user name is irrelevant; the kernel doesn't even know about user names.
Android's permission mechanism is identical at the kernel level but completely different at the application level. Android has a root user (UID 0), just like any other system based on a Linux kernel. Android doesn't have user accounts though, and on most setups doesn't allow the user (as in the human operating and owning the device) to perform actions as the root user. A “rooted” Android is a setup that does allow the device owner/user to perform actions as root.
How setuid works
A setuid executable runs as the user who owns the executable. For example, su is setuid and owned by root, so when any user runs it, the process running su runs as the root user. The job of su is to verify that the user that calls it is allowed to use the root account, to run the specified command (or a shell if no command is specified) if this verification succeeds, and to exit if this verification fails. For example, su might ask the user to prove that they know the root password.
In more detail, a process has three user IDs: the effective UID, which is used for security checks; the real UID, which is used in a few privilege checks but is mainly useful as a backup of the original user ID, and the saved user ID which allows a process to temporarily switch its effective UID to the real user ID and then go back to the former effective UID (this is useful e.g. when a setuid program needs to access a file as the original user). Running a setuid executable sets the effective UID to the owner of executable and retains the real UID.
Running a setuid executable (and similar mechanisms, e.g. setgid) is the only way to elevate the privileges of a process. Pretty much everything else can only decrease the privileges of a process.
Beyond traditional Unix
Until now I described traditional Unix systems. All of this is true on a modern Linux system, but Linux brings several additional complications.
Linux has a capability system. Remember how I said that the kernel has many checks where only processes running as user ID 0 are allowed? In fact, each check gets its own capability (well, not quite, some checks use the same capability). For example, there's a capability for accessing raw network sockets, and another capability for rebooting the system. Each process has a set of capabilities along side its users and groups. The process passes the check if it is running as user 0 or if it has the capability that corresponds to the check. A process that requires a specific privilege can run as a non-root user but with the requisite capability; this limits the impact if the process has a security hole. An executable can be setcap to one or more capabilities: this is similar to setuid, but works on the process's capability set instead of the process's user ID. For example, ping only needs raw network sockets, so it can be setcap CAP_NET_RAW instead of setuid root.
Linux has several security modules, the best known being SELinux. Security modules introduce additional security checks, which can apply even to processes running as root. For example, it's possible (not easy!) to set up SELinux so as to run a process as user ID 0 but with so many restrictions that it can't actually do anything.
Linux has user namespaces. Inside the kernel, a user is in fact not just a user ID, but a pair consisting of a user ID and a namespace. Namespaces form a hierarchy: a child namespace refines permissions within its parent. The all-powerful user is user 0 in the root namespace. User 0 in a namespace has powers only inside that namespace. For example, user 0 in a user namespace can impersonate any user of that namespace; but from the outside all the processes in that namespace run as the same user.
Best Answer
Giving a process the right to change its UID is equivalent to giving it the right to run as root, since it can just change its UID to root. So even if you did that, there'd be no benefit.
You can improve the security of this setup by reducing the risk that the program can be tricked into doing something unintended as root. This won't reduce the impact of a complete compromise (if an attacker is able to make your program run arbitrary code), but it can reduce the impact of a partial compromise (where the attacker is only able to convince your program to do some specific thing, e.g. to read a file and display it). To reduce the risk, reduce the window of time during which the program runs as root. Make it run as a dedicated user, but keep its real UID set to 0. When the program needs to be root, it temporarily switches its effective UID to 0, and then switches back.
Note that this is not really practical in a multithreaded program since the user ID is for a whole process, not per thread.
For better robustness and security, implement privilege separation: split your program into two processes, one that runs as root and one that runs as a dedicated user. The process that runs as root should do as little as possible. It takes requests from the other process, validates them to the extent possible, and performs them. Then, even if the other process is buggy, the most it can do is issue valid requests.
For simple needs, rather than write a program that runs as root, you may be able to have a sudo rule that directly allows the main program to run commands as a third user. Instead of relying on an intermediate root program to perform validation, you can embed the validation in the sudo rule, e.g.
This allows
myapp_userto run a command as any user, but the command has to bevalidate_and_run(with any arguments).