Linux – Saving data from kernel module into userspace

filesystemsinterruptkernellinuxraspberry pi

I have been playing around kernel programming for a while and want to create this simple data acquiring interface with some custom hardware. For portability and reusability, I do the whole thing on my Raspberry Pi.

The challenging part of the project is having a high speed ADC (parallel) connected to GPIO's and having a kernel module that uses hardware interrupt from ADC to acquire each sample and store it inside a buffer which is then accessible via chardevice.

My current setup (that works) is as follows:

  • I have a userspace C program that is controlling my hardware through SPI. If I send a required command, it starts acquiring analogue data and sends them to the ADC.
  • Whenever ADC finishes conversion, it pusts corresponding signal to 'low' on a GPIO and I get interrupt inside the kernel module (bound to that GPIO). The ISR collects the value of 12 other GPIO's (it's a 12-bit ADC) and puts it into a buffer that is then accessed through /dev/mydevice.
  • I have another separate userspace program that runs a never-ending while loop, reading from /dev/mydevice and in turn writes into 'out_data.dat' (an userspace file).
  • With this crude setup (2 userspace programs and kernel module loaded) I can write over 130 000 samples into my file per second (without missing anything).

I now want to see how much faster I can make it, there are 2 things to consider:

  1. Is the setup I have outline above the 'usual' way how something like this would be done? I read everywhere that direct file I/O is not advised from kernel so I am not doing it. Surely though, it should be possible to write it into some "permanent" location during the ISR. This seems to me like a common problem, trying to get data from some hardware into computer using interrupts.

  2. Without changing my setup above, is there any way how to disable other interrupts to make it as smooth as possible? During the data acquisition I do not really need anything, only some sort of a way how to stop it. Any other interrupts (wireless, monitor refresh etc…) can be disabled as data acquisition is only to be run for a few minutes. Afterwards, everything will resume and more demanding python code can be run to analyze and visualize the data (at least that's my simple view of it).

Best Answer

For the userspace data collection program, what is wrong with an infinite loop? As long as you are using the poll system call, it should be efficient: https://stackoverflow.com/questions/30035776/how-to-add-poll-function-to-the-kernel-module-code/44645336#44645336 ?

Permanent data storage

I'm not sure what is the best way to do it, why don't you just write to a file from userland on the poll? I suppose your concern is that if too much data arrives, data would be lost, is that so?

But I doubt the limiting factor would be kernel to userland communication in that case, but rather the slowness of the permanent storage device, so doing it on userland won't make any difference I think. In any case, the kernel only solution has a high profile question at: https://stackoverflow.com/questions/1184274/how-to-read-write-files-within-a-linux-kernel-module and I don't think you will get a better solution here.

Disable interrupts

Are you sure that it would make any difference, especially considering that the bottleneck is likely going to be? I would expect that if your device is actually producing a large number of interrupts, then those would dominate any other interrupts in any case. Is it worth risking messing up the state of other hardware? Do the specs of your hardware device suggest that it could physically provide a much larger data bandwidth that what you currently have?

I don't know how to do it myself, but if you want an answer, your best bet is to make a separate question with title "How to disable all interrupts from a Linux kernel module?". LDD2 mentions the cli() function http://www.xml.com/ldd/chapter/book/ch09.html but it seems that it was deprecated: https://notes.shichao.io/lkd/ch7/#no-more-global-cli That text then suggests local_irq_disable and local_irq_save.

I would also try to hack it up with whatever method you find to disable the interrupts, and see if it gets any more efficient before looking further if a nice method exists.

On an emulator, a quick:

static int myinit(void)
{
    pr_info("hello init\n");
    unsigned long flags;
    local_irq_save(flags);
    return 0;
}

fails with:

returned with disabled interrupts

apparently coming from v4.16 do_one_initcall, so there is a specialized error handling for that!

I then tried naively doing it from a worker thread:

static int work_func(void *data)
{
    unsigned long flags;
    local_irq_save(flags);
    return 0;
}

static int myinit(void)
{
    kthread = kthread_create(work_func, NULL, "mykthread");
    wake_up_process(kthread);
    return 0;
}

but still then I can't observe any effect, so the interrupts must be being enabled by something else, as can be inferred from:

watch -n 1 grep i8042 /proc/interrupts

which keeps updating tty or muse / keyboard interrupts.

Same from other entry points such as fops, or if I try a raw asm("cli"). We will need some more educated approach.

Related Question