Yes, the reason that essential components (such as mm) cannot be loadable modules is because they are essential -- the kernel will not work without them.
I can't find any references claiming the effects of memory fragmentation with regard to loadable modules is significant, but this part of the LLKM how-to might be interesting reading for you.
I think the question is really part and parcel of the issue of memory fragmentation generally, which happens on two levels: the fragmentation of real memory, which the kernel mm subsystem manages, and the fragmentation of virtual address space which may occur with very large applications (which I'd presume is mostly the result of how they are designed and compiled).
With regard to the fragmentation of real memory, I do not think this is possible at finer than page size (4 KB) granularity. So if you were reading 1 MB of virtually contiguous space that is actually 100% fragmented into 1024 pages, there may be 1000 extra minor operations involved. In that bit of the how-to we read:
The base kernel contains within its prized contiguous domain a large
expanse of reusable memory -- the kmalloc pool. In some versions of
Linux, the module loader tries first to get contiguous memory from
that pool into which to load an LKM and only if a large enough space
was not available, go to the vmalloc space. Andi Kleen submitted code
to do that in Linux 2.5 in October 2002. He claims the difference is
in the several per cent range.
Here the vmalloc space, which is where userspace applications reside, would be that which is potentially prone to fragment into pages. This is simply the reality of contemporary operating systems (they all manage memory via virtual addressing). We might infer from this that virtual addressing could represent a performance penalty of "several percent" in userland as well, but in so far as virtual addressing is necessary and inescapable in userland, it is only in relation to something completely theoretical.
There is the possibility for further compounding fragmentation by the fragmentation of a process's virtual address space (as opposed to the real memory behind it), but this would never apply to kernel modules (whereas the last paragraph apparently could).
If you want my opinion, it is not worth much contemplation. Keep in mind that even with a highly modular kernel, the most used components (fs, networking, etc) will tend to be loaded very early and remain loaded, hence they will certainly be in a contiguous region of real memory, for what it is worth (which might be a reason to not pointlessly load and unload modules).
Best Answer
No part of the Linux kernel can be paged out, even parts that come from modules.
A kernel module can be loaded and (if the module supports it) can be unloaded. This always happens from an explicit request from a userland process with the
init_moduleanddelete_modulesystem calls (normally, via theinsmodormodprobeutilities for loading, and viarmmodfor unloading).Once a module is loaded, it's part of the kernel, like any other part of the kernel. In particular, there's no way to isolate the memory used by a specific module. The kernel keeps tracks of which part of the memory contains a specific module's code, but not where a module may have stored data. A module can modify any of the kernel's data structures, after all.
A module can potentially add some code to any kernel subsystem. Most modules are hardware drivers, but some aren't (e.g. they can provide security functionality, filesystems, networking functionality, etc.). If data or code used by a module could be swapped out, then the rest of the kernel would have to load it when required, which would complicate the design of the system a lot. The kernel would also need to ensure that no part of the memory that's swapped out is ever necessary to swap it back in, which is difficult. What if the swap is in a swap file on a network filesystem, and the module provides firewall functionality that is involved in communicating with the server that stores the file?
It's possible to completely unload a module because it's the module's job to provide code that ensures that the module is not needed for anything. The kernel runs the module's exit function, and only unloads the module if that function reports that the module can be safely unloaded. The exit function must free any remaining data memory that is “owned” by the module (i.e. data that the module needs, but no other part of the kernel needs), and must verify that no code in the module is registered to be called when something happens. There's no way to save a module's data to swap: a module can only be removed from RAM if it has no data left.