A user program mess with kernel space

From the operating system's perspective, user space is the execution of code from a process, and kernel space is the execution of code from the kernel. A transition from user space to kernel space is the entry into a system call, and a transition from kernel space to user space is the return from a system call.

From the processor's perspective, user space and kernel space are privilege levels. “Ring 3” and “ring 0” are the names used on Intel x86 processors (there are rings 1 and 2, but most unices don't use them, because there isn't a lot of use for them). Other CPU types use different names, for example user mode and privileged (or system) modes on ARM.

Transition between modes is done by instructions that both change the processor mode and jump to a different address; the detail of these instructions is very much dependent on the CPU type. In addition to switching the processor mode and jumping to a different address, the mode transition instructions typically perform some other tasks such as swapping the value of a few registers (again, this is very CPU-dependent).

A transition from user mode to kernel mode has to jump to a fixed address, which has been set by the kernel. In order to maintain the security of the system, code in user mode cannot be allowed to switch the processor to kernel mode except to invoke a piece of code (the system call entry point) which has been specially crafted to work securely no matter what the user mode code has been doing. The system call entry point first saves register values into a designated memory area, and performs any other necessary bookkeeping, then it reads the system call parameters and dispatches the system call to the appropriate function. Depending on the processor type, the bookkeeping may involve setting the MMU tables appropriately. The kernel may decide at any point to pause the calling process and schedule another process.

A transition from kernel mode to user mode can happen anywhere the kernel deems fit. The kernel restores the saved registers, the MMU configuration and anything else that is necessary, and writes the system call's return value, and finally jumps back to the instruction in the process after the system call.

In the case of Linux, a task (kernel internal idea of a thread; threads can share resources, like memory and open files; some run only inside the kernel) can run in userland, or (it's thread of execution) can transfer into the kernel (and back) to execute a system call. A user thread can be highjacked temporarily to execute an interrupt (but that isn't really that thread running).

That a process is a "system process" or a regular user process is completely irrelevant in Unix, they are handled just the same. In Linux case, some tasks run in-kernel to handle miscellaneous jobs. They are kernel jobs, not "system processes" however.

One big caveat: Text books on complex software products (compilers and operating systems are particularly egregious examples) tend to explain simplistic algorithms (often ones that haven't been used in earnest for half a century), because real world machines and user requirements are much too complex to be handled in some way that can be described in a structured, simple way. Much of a compiler is ad-hoc tweaks (particularly in the area of code optimization, the transformations are mostly the subset of possibilities that show up in practical use). In the case of Linux, most of the code is device drivers (mentioned in passing as device-dependent in operating system texts), and of this code a hefty slice is handling misbehaving devices, which do not comply to their own specifications, or which behave differently between versions of "the same device". Often what is explained in minute detail is just the segment of the job that can be reduced to some nice theory, leaving the messy, irregular part (almost) completely out. For instance, Cris Fraser and David Hanson in their book describing the LCC compiler state that typical compiler texts contain mostly explanations on lexical analysis and parsing, and very little on code generation. Those tasks are some 5% of the code of their (engineered to be simple!) compiler, and had negligible error rate. The complex part of the compiler is just not covered in standard texts.