Linux – How to (really) disable NCQ in Linux

driverskernellinuxsata

I implemented my own Serial-ATA Host-Bus-Adapter (HBA) in VHDL and programmed it onto a FPGA. A FPGA is chip which can be programmed with any digital circuit. It's also equipped with serial transceivers to generate high speed signals for SATA or PCIe.

This SATA controller supports SATA 6 Gb/s line rates and uses ATA-8 DMA-IN/OUT commands to transfer data in up to 32 MiB chunks to and from the device. The design is proven to work at maximum speed (e.g. Samsung SSD 840 Pro -> over 550 MiB/s).

After some tests with several SSD and HDD devices, I bought a new Seagate 6 TB Archive HDD (ST6000AS0002). This HDD reaches up to 190 MiB/s read performance, but only 30 to 40 MiB/s write performance!

So I dug deeper and measured the transmitted frames (yes that's possible with a FPGA design). As far as I can tell, the Seagate HDD is ready to receive the first 32 MiB of a transfer in one piece. This transfer happens at maximum line speed of 580 MiB/s. After that, the HDD stalls the remaining bytes for over 800 ms! Then the HDD is ready to receive the next 32 MiB and stalls again for 800 ms. All in all an 1 GiB transfer needs over 30 seconds, which equals to circa 35 MiB/s.

I assume that this HDD has a 32 MiB write cache, which is flushed in between the burst cycles. Data transfers with less than 32 MiB don't show this behavior.

My controller uses DMA-IN and DMA-OUT command to transfer data. I'm not using the QUEUED-DMA-IN and QUEUED-DMA-OUT command, which are used by NCQ capable AHCI controllers. Inplementing AHCI and NCQ on a FPGA platform is very complex and not needed by my application layer.

I would like to reproduce this scenario on my Linux PC, but the Linux AHCI driver has NCQ enabled by default. I need to disable NCQ, so I found this website describing how to disable NCQ, but it doesn't work.

The Linux PC still reaches 190 MiB/s write performance.

> dd if=/dev/zero of=/dev/sdb bs=32M count=32
1073741824 bytes (1.1 GB) copied, 5.46148 s, 197 MB/s

I think there is a fault in the article from above: Reducing the NCQ queue depth to 1 does not disable NCQ. It just allows the OS the use only one queue. It can still use QUEUED-DMA-** commands for the transfer. I need to realy disable NCQ so the driver issues DMA-IN/OUT commands to the device.

So here are my questions:

  1. How can I disable NCQ?
  2. If NCQ queue depth = 1, is Linux's AHCI driver using QUEUED-DMA-** or DMA-** commands?
  3. How can I check if NCQ is disable, because changing /sys/block/sdX/device/queue_depth is not reported in dmesg?

Best Answer

Thanks to @frostschutz, I could measure the write performance in Linux without NCQ feature. The kernel boot parameter libata.force=noncq disabled NCQ completely.

Regarding my Seagate 6TB write performance problem, there was no change in speed. Linux still reaches 180 MiB/s.

But then I had another idea:
The Linux driver does not use transfers of 32 MiB chunks. The kernel buffer is much smaller, especially if NCQ with 32 queues is enabled (32 queues * 32 MiB => 1 GiB AHCI buffer).

So I tested my SATA controller with 256 KiB transfers and voilà, it's possible to reach 185 MiB/s.

So I guess the Seagate ST6000AS0002 firmware is not capable of handling big ATA burst transfers. The ATA standard allows up to 65.536 logical blocks, which equals 32 MiB.

SMR - Shingled Magnetic Recording

Another possibility for the bad write performance could be the shingled magnetic recording technique, which is used by Seagate in these archive devices. Obviously, I triggered a rare effect with my FPGA implementation.

Related Question