I use the CUDA toolkit to perform some computations on my Nvidia GPUs. How to kill all processes that use a given GPU? (killing at once, i.e. without having to manually type the PIDs behind kill -9.)
E.g. killing all processes using GPU 2:
Best Answer
Following the Unix philosophy, you have a tool that lists processes using a given GPU, and a tool that kills processes. Combine them using shell constructs and text processing tools.
For example, to kill all the processes using GPU 2, you can execute the following command:
I "kind of" know a little bit about this due to running both a Plex server and a ML instance on the same bare metal server a few years back. CUDA 5.5 added the option to set stream priories at the driver level enabling a end-user to schedule priorities just like other activities in Linux. At a GUI level Nvidia added "performance modes" in the settings around 2016. Here's an article I found detailing this: http://ubuntuhandbook.org/index.php/2016/04/switch-intel-nvidia-graphics-ubuntu-16-04/ . I don't know if this will help at all but figured I'd share what helped me.
Best Answer
Following the Unix philosophy, you have a tool that lists processes using a given GPU, and a tool that kills processes. Combine them using shell constructs and text processing tools.
For example, to kill all the processes using GPU 2, you can execute the following command:
or