Linux – Prevent Linux kernel from setting USB device configuration

driverslinux-kernelproprietary-driversusbusb device

I'm trying to write a Linux driver for an existing USB device, based on captured USB communication.

The device has multiple configurations. Unfortunately, it seems like the device doesn't follow the USB specification: Only the first Set Configuration Request works. The device locks and eventually crashes on issuing a second Set Configuration Request. USB reset or resetting the configuration (setting it to 0) also doesn't help anything.

Now, it seems the Linux USB Core for whatever reason decides to set the device's configuration to the wrong value (it just picks the first one), without me having any chance to step in before it does so. I already tried it from a kernel module and from a user space libusb driver.

From reading through the kernel source code, it seems like the function that selects the configuration is usb_choose_configuration() in the generic driver at /drivers/usb/core/generic.c. I can see that the function could be skipped if usb_device_is_owned() returned true, but I have no idea how I could influence the result of that function.

I'm hoping I won't have to recompile the whole kernel just to add a USB driver.

Thus, here are my questions:

  • How could I prevent the system from setting a configuration before handing control to my driver?
  • It seems in recent kernel versions, usbcore is a builtin module that cannot be replaced. Is there any other way I could override the usb_choose_configuration function in the generic driver (which seems to be part of usbcore)?
  • How could I make the device owned, so that usb_device_is_owned() returns true already when the device is attached?

Best Answer

It seems there is a way to prevent the system from trying to set the device's configuration, and it even works from user space. I stumbled upon the commit that added this functionality to the kernel, and luckily it also includes some sample code.

User-space programs can declare ownership of a particular port of a USB hub via the device file system, resulting in usb_device_is_owned() returning true.

The trick seems to be:

unsigned int port = 2; // Just as an example
// Send request 24 of type 'U' (USB), which returns an unsigned int
unsigned int ioctl_id = _IOR('U', 24, unsigned int);
// fd is a file descriptor to the hub's file in the devfs
ioctl(fd, ioctl_id, &port);

Information on some of the ioctl requests of the USB subsystem is described in the kernel documentation. The full list can be found in the kernel source. The #defines are here.

Interestingly, the system still sends Set Configuration Requests for configurations 0 (reset configuration).

Related Question