GPU usage per process on a Linux machine (CUDA)

I "kind of" know a little bit about this due to running both a Plex server and a ML instance on the same bare metal server a few years back. CUDA 5.5 added the option to set stream priories at the driver level enabling a end-user to schedule priorities just like other activities in Linux. At a GUI level Nvidia added "performance modes" in the settings around 2016. Here's an article I found detailing this: http://ubuntuhandbook.org/index.php/2016/04/switch-intel-nvidia-graphics-ubuntu-16-04/ . I don't know if this will help at all but figured I'd share what helped me.

Best of luck!

GPUs aren't yet "native" to the kernels... actually, they are quite "foreign" to the linux kernels, as they are (especially the CUDA, perhaps less so the OpenCL related stuff) relying on proprietary drivers etc. to make the stuff run inside the GPUs.

So no, don't expect a "simple" time/timex to show those counters, until somebody writes that, but what I've seem thus far based on the coding needed, I don't expect that to be available soon unless you code it into the applications you are calling that does the CUDA/OpenCL stuff for you.