Illumos is not a full replacement to OSOL and I don't think it will be in the future since it's intended to be a base from which others can build a distribution. But check the Nexenta OS, this system is heavily based on OSOL and they are one of the main sponsors behind the Illumos project. Although I haven't used it personally and I wouldn't know whether it has what you need.
If you choose to make an operating system based on the Linux kernel then you have to distribute it for free under the GPL License,
That's not quite true. You can make an OS based on the Linux kernel with no constraint whatsoever, as long as you keep it for yourself. If you distribute an OS based on the Linux kernel, then you have to distribute the source code of the kernel (or any other part where you've used code from the Linux kernel). You don't have to distribute the rest. For example, most Linux distributions include some proprietary software; the GNU GPL doesn't constrain software that is distributed together with software covered by the GPL.
but if you choose to make an OS based on the Unix kernel
There is no such thing as “the Unix kernel” — not anymore. There are many Unix kernels, of which the Linux kernel is one. Some of them are based on the original Unix from Bell Labs (Solaris, HP-UX), others are not (*BSD, Linux, MINIX).
(example: an OS based on FreeBSD) do you have permission to make it closed-source and to take copyrights making it a proprietary software distributing it non-free?
FreeBSD code comes under a BSD license which is extremely liberal and includes the right to distribute proprietary software based on the BSD-licensed software. FreeBSD is not derived from the original Unix product, which was a commercial product. (BSD was originally companion software for a commercial Unix, and eventually they rewrote all the parts under a free license.)
So if somebody chose to make an OS based on FreeBSD can they sell it as their own modified version, taking copyrights or something like that? This question arose because I know that Mac OS X is based on FreeBSD and it should have used FreeBSD licenses, and OS X is a non-free, closed-source proprietary software.
Yes, the FreeBSD license allows that.
So, can you do that with Unix? Or does Apple have some sort of "agreement"?
You can't do that with the original Unix product, but that hasn't existed as a product for a long time (and there never really was a single Unix product except at the very beginning). You can do that with the Linux kernel (and with the GNU userland, too), as long as you distribute the sources for the GPL parts that you distribute (including your modifications if you modified the sources); you can keep the source of independent components (separate programs and libraries) for yourself. You can do that with FreeBSD, with basically no constraint.
Best Answer
Three reasons:
First, being certified as a Unix says nothing about your licensing, just your compliance to the established standards for being Unix.
Second, because being Unix has nothing to do with your licensing, and everything to do with your being like Unix, an originally proprietary system, and one with a long legacy.
Finally, don't mistake licensing: BSD code can subsequently be closed, and there is no duty of disclosure or source availability. The BSD license is more free on one axis, and less free on another, because of that.