I temporarily disable ECC memory protection on a NVIDIA K20m (device 0 in my node) and now I cannot bring it back to work again.
Before that it was working properly with ECC enabled.
So, here is what I did:
I disabled ECC with
nvidia-smi -i 0 --ecc-config=0
and rebooted. When it came up it showed 100% GPU utilization and it wouldn't start any kernels (it actually already failed when creating the context). The reason was a double bit error. I reset it with
nvidia-smi -i 0 --reset-ecc-errors=0
and rebooted the node. After the reboot the device utilization was 0% and I could start jobs as usual. A few hours later the device showed again a 100% GPU utilization. This time it didn't report a double bit error (not even a single bit error). However, since I couldn't run any job I rebooted the node and it came up with 100% GPU utilization, I can't use it, but reports no bit errors. What's the matter this it?
GPU 0000:02:00.0
Product Name : Tesla K20m
Display Mode : Disabled
Persistence Mode : Enabled
Driver Model
Current : N/A
Pending : N/A
Serial Number : 0324512044699
GPU UUID : GPU-9bfe1aba-1628-a406-3ed5-2af49462a997
VBIOS Version : 80.10.11.00.0B
Inforom Version
Image Version : 2081.0208.01.07
OEM Object : 1.1
ECC Object : 3.0
Power Management Object : N/A
GPU Operation Mode
Current : Compute
Pending : Compute
PCI
Bus : 0x02
Device : 0x00
Domain : 0x0000
Device Id : 0x102810DE
Bus Id : 0000:02:00.0
Sub System Id : 0x101510DE
GPU Link Info
PCIe Generation
Max : 2
Current : 2
Link Width
Max : 16x
Current : 16x
Fan Speed : N/A
Performance State : P0
Clocks Throttle Reasons
Idle : Not Active
User Defined Clocks : Not Active
SW Power Cap : Not Active
HW Slowdown : Not Active
Unknown : Not Active
Memory Usage
Total : 4799 MB
Used : 12 MB
Free : 4787 MB
Compute Mode : Default
Utilization
Gpu : 100 %
Memory : 0 %
Ecc Mode
Current : Enabled
Pending : Enabled
ECC Errors
Volatile
Single Bit
Device Memory : 0
Register File : 0
L1 Cache : 0
L2 Cache : 0
Texture Memory : 0
Total : 0
Double Bit
Device Memory : 0
Register File : 0
L1 Cache : 0
L2 Cache : 0
Texture Memory : 0
Total : 0
Aggregate
Single Bit
Device Memory : 0
Register File : 0
L1 Cache : 0
L2 Cache : 0
Texture Memory : 0
Total : 0
Double Bit
Device Memory : 0
Register File : 0
L1 Cache : 0
L2 Cache : 0
Texture Memory : 0
Total : 0
Temperature
Gpu : 30 C
Power Readings
Power Management : Supported
Power Draw : 49.51 W
Power Limit : 225.00 W
Default Power Limit : 225.00 W
Min Power Limit : 150.00 W
Max Power Limit : 225.00 W
Clocks
Graphics : 758 MHz
SM : 758 MHz
Memory : 2600 MHz
Applications Clocks
Graphics : 705 MHz
Memory : 2600 MHz
Max Clocks
Graphics : 758 MHz
SM : 758 MHz
Memory : 2600 MHz
Compute Processes : None
Best Answer
Is my GPU dying?
I say it's already dead. It's no longer reporting bit errors because you turned off the thing that detects them. (ECC detects more than it can correct.) However, it may be the memory on the card (or the physical card itself) that's developed a fault.
There's two other suspects before I'd chuck it in the "recycle" bin: cooling, and power supply. Cooling is easy enough to check; power, not so much.