Linux – What’s the difference between ioremap and file operation mmap

OK, it looks like the subject of this question is a furphy ... memory mapped I/O (done correctly) will be as fast as the processor can do it for the hardware being accessed and there will be no overhead for doing this from user mode as opposed to kernel mode (i.e. there is no "write from the userspace to kernel").

However, you still have to think about what is happening when you do a read from or a write to an address (this is where the question moved on to). In most architectures, there are two mappings - the virtual to the physical, and the physical to the device. The first is set up in the virtual memory hardware and the second is set up in the memory controller.

In addition to the mappings, all accesses usually go via cache hardware, so you have to decide whether you want the accesses to be cached or not. If the underlying device being accessed is RAM of some sort, then you usually want accesses to be cached. For other types of devices, generally you don't.

There may be a lot of other things to think about (e.g. whether the VM mapping is resident in the VM hardware, the width and timing of accesses, priority, permissions, etc) but cache is the first.

In @Karthik's case, because he had not turned off cache in his mapping, depending on the type of cache, either an entire cache line was being written when he wrote to the address (write-through), or the write was being delayed (write-back) (if you want some nitty gritty about cache, try this).

To answer the specific (follow up) questions, once the virtual address mapping is done and the cache has done its job, the access goes to the memory controller - this hardware decides which bus and/or device is being accessed and does the "right thing" for that hardware, usually involving asserting a chip select and/or a write enable signal, maybe copying a part or all of the physical address onto address lines, maybe some setup timing, etc.

... and the best way to debug this stuff is to have an analyser of some sort connected to your device or bus, or if this is too difficult/expensive, there might be some support for debugging in the memory controller.

One other minor but important point ... take note of the declaration of blink_mem in the code above - the volatile type qualifier is very important. It tells the compiler not to muck with accesses to the address. In addition to this, you should be aware of any special pipeline instructions to do with memory accesses (check out the eieio instruction in powerpc - someone has a sense of humour :-)

Finally, just to re-iterate what was said in the comments, which turned out to be the real answer to the question, when calling remap_pfn_range() you turn off cache by modifying the page protections specified in the last argument (prot) using the pgprot_noncached() macro. Also read this and this and particularly this. Cheers!

1. The kernel mapping exists primarily for the kernel’s purposes, not user processes’. From the CPU’s perspective, any physical memory address which isn’t mapped as a linear address might as well not exist. But the CPU does need to be able to call into the kernel: to service interrupts, to handle exceptions... It also needs to be able to call into the kernel when a user process issues a system call (there are various ways this can happen so I won’t go into details). On most if not all architectures, this happens without the opportunity to switch page tables — see for example SYSENTER. So at minimum, entry points into the kernel have to be mapped into the current address space at all times.

2. Kernel allocations are dynamic, but the address space isn’t. On 32-bit x86, various splits are available, such as the 3/1 GiB split shown in your diagram; on 64-bit x86, the top half of the address space is reserved for the kernel (see the memory map in the kernel documentation). That split can’t move. (Note that libraries are loaded into user space. Kernel modules are loaded into kernel space, but again that only changes the allocations, not the address space split.)

3. In user mode, there is a single mapping for the kernel, shared across all processes. When a kernel-side page mapping changes, that change is reflected everywhere.

   When KPTI is enabled, the kernel has its own private mappings, which aren’t exposed when running user-space code; so with KPTI there are two mappings, and changes to the kernel-private one won’t be visible to user-space (which is the whole point of KPTI).

   The kernel memory map always maps all the kernel (in kernel mode when running KPTI), but it’s not necessarily one-to-one — on 64-bit x86 for example it includes a full map of physical memory, so all kernel physical addresses are mapped at least twice.