Ubuntu – Pycharm debugger does not work with pytorch and deep learning

cudadebuggerpycharmpythonUbuntu

This question is somewhere in between Stackoverflow and Superuser – in my opinion at least, so feel free to point me to SO if this is the wrong place (in your opinion ;)).

Problem:

If I place a breakpoint in my python code the debugger stops. If I want to step into the next line, it will do so.
Unless it happens after a enumerate(dataloader). I can place a breakpoint after enumerate(dataloader) and the debugger stops as well. However I can not step then.
Same happens if I place the breakpoint before enumerate(dataloader) and step over the enumerate line. It won't work for the line after enumerate(..).

The interesting fact is, that it worked until recently – like a week ago

# Breakpoints work, stepping over works
...
for epoch in range(num_epochs):
    for i, data in enumerate(dataloader, 0):
        # If I break before this line and I step until the next, I can not step afterwards
        netD.zero_grad()
        # Breakpoints work, stepping does not

What did I try to solve this:

  • I uninstalled PyCharm and reinstalled it.
  • I tried using the Community edition.
  • I tried running both versions with and without the cython debugger extension.
  • I've checked which things I installed after it stopped working and removed them.
  • I've checked numerous posts on Jetbrains, SO and the Pytorch forums. Found some that looked interesting, but in fact they didn't fit my problem.
  • I tried using different CUDA versions and cudnn versions.

Nothing worked so far … I don't really want to reinstall my OS, which would most likely help, but yeah, for obvious reasons that is not what I want.
Has anyone EVER experienced this kind of behavior and can help me? Thanks in advance. Feel free to ask for additional details. I'll try to provide as many details as possible

I will list some of the details of my setup:

OS:

  • Intel® Core™ i7-7700K CPU @ 4.20GHz × 8
  • Nvidia RTX 2080 TI
  • NVIDIA-SMI 430.26 Driver Version: 430.26 CUDA Version: 10.2
  • Ubuntu 18.04

Software:

  • PyCharm Community and EDU 2019.2
  • (Ana)conda
  • cudnn installed as well
  • Two conda envs for python 3.6 and python 3.7 with pytorch and everything it needs (numpy, scipy, matplotlib, etc)

nvcc:

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2019 NVIDIA Corporation
Built on Wed_Apr_24_19:10:27_PDT_2019
Cuda compilation tools, release 10.1, V10.1.168

Best Answer

I had the same problem for weeks and just managed to fix it! The solution was to set num_workers to 0 during creation of the DataLoader object.

Related Question