Having such a huge swap nowadays is often a bad idea. By the time the OS swapped just a few GB of memory to swap, your system had already crawled to death (like what you saw)
It's better to use zram with a small backup swap partition. Many OSes like ChromeOS, Android and various Linux distros (Lubuntu, Fedora) have enabled zram by default for years, especially for systems with less RAM. It's much faster than swap on HDD and you can clearly feel the system responsiveness in this case. Less so on an SSD, but according to the benchmark results here it still seems faster even with the default lzo algorithm. You can change to lz4 for even better performance with a little bit less compression ratio. It's decoding speed is nearly 5 times faster than lzo based on official benchmark
In fact Windows 10 and macOS also use similar pagefile compression techniques by default
There's also zswap although I've never used it. Probably worth a try and compare which one is better for your usecases
After that another suggestion is to reduce the priority of those IO-bound processes and possibly leave a terminal running on higher priority so that you can run commands on it right away even when the system is on a high load
Further reading
Encouraged by the comment from @dirkt, I looked better at the drivers and found the PCI MSI interrupts that correspond to these DMA transactions.
The driver enables these interrupts with a call
pci_enable_msix(.., msixTable,..)
that sets up the struct msix_entry msixTable[MAXMSIX]. Then it asings them to the handler static irqreturn_t irqHandler() by calling request_irq() in a loop:
request_irq(msixTable[interrupt].vector, irqHandler, 0, devName,...)
The handler just counts the interrupts in a local int array. These counters are exported in the /proc/<devName> file that this driver creates for diagnostics etc. In fact, the proc file is from where I started the search for the interrupts.
But there is a better way: the /proc/interrupts file. The enabled MSI-X interrupts show up there in lines like these:
$ cat /proc/interrupts
CPU0 CPU1 ... CPU5 CPU6 CPU7
66: 0 0 ... 0 0 0 IR-PCI-MSI-edge <devName>
67: 0 0 ... 0 0 0 IR-PCI-MSI-edge <devName>
68: 33 0 ... 0 0 0 IR-PCI-MSI-edge <devName>
69: 0 0 ... 0 0 0 IR-PCI-MSI-edge <devName>
70: 0 0 ... 0 0 0 IR-PCI-MSI-edge <devName>
71: 0 0 ... 0 0 0 IR-PCI-MSI-edge <devName>
72: 0 0 ... 0 0 0 IR-PCI-MSI-edge <devName>
73: 0 0 ... 0 0 0 IR-PCI-MSI-edge <devName>
And one more way is to find the PCI address of the card in the lspci output and to check the interrupts assigned to the card in the /sys directory:
$ ls /sys/bus/pci/devices/0000:17:00.0/msi_irqs
66 67 68 69 70 71 72 73
# but these are empty
$ cat /sys/bus/pci/devices/0000:17:00.0/irq
0
The interrupt number 68 fires up by the end of the transactions. The interrupt handlers have a static tracepoint irq:irq_handler_entry in Linux. The tracepoint parameters in /sys/kernel/debug/tracing/events/irq/irq_handler_entry/format have the interrupt number in the int irq field. Hence, this interrupt can be traced with the standard Linux facilities by this tracepoint with a filter condition:
# setup the ftrace
trace-cmd start -e irq:irq_handler_entry -f "irq == 68"
# for live stream
cat /sys/kernel/debug/tracing/trace_pipe
# or just
trace-cmd stop
trace-cmd show
trace-cmd reset
# with perf
perf record -e "irq:irq_handler_entry" --filter "irq == 68"
With this, one thing that is still worth confirming is that these interrupts are essential to the DMA, to be sure that I monitor something relevant to the system instead of just a handy counter for the proc file that might not be implemented in another situation. But I could not spot that any other relevant interrupts by watching at how they increment in the /proc/interrupts. There are interrupts for the devices dmar[0123] that seem like something about DMA, but they have never incremented. And that is to be expected, as in this case the DMA engine must be implemented as an FPGA core in the PCI card itself.
Best Answer
You can use gdb to access the memory of a process.
Also, you should have a look at the "/proc" filesystem - it contains pseudo files for every process; some of them may contain interesting information