The process will be killed without any error message

[BJUT-AIVBD-1]

Hi Fabian,

Thanks for sharing the nnU-net! I'm trying to train a 2D model with massachusetts road segmentation dataset. However, when the training process reaches about the 4th epoch, the terminal will display "killed" without any error message. Then the training process will be interrupted, and a large number of processes (about 8G) will remain in GPU memory. I tried to reduce batch_size to 1, and the same thing happened at the 10th epoch. Why does this happen? My computer uses Ubuntu with a 2080Ti, and the CUDA and CUDNN versions are 10.2.89 and 7.6.5 respectively.

[FabianIsensee]

Read this: nnU-Net training (2D U-Net): High (and increasing) system RAM usage, OOM
https://github.com/MIC-DKFZ/nnUNet/blob/master/documentation/common_problems_and_solutions.md#nnu-net-training-2d-u-net-high-and-increasing-system-ram-usage-oom

[BJUT-AIVBD-1]

Thank you, I updated CUDA and CUDNN versions to 11.0.194 and 8.0.5 respectively, then I recompile pyTorch, and it worked.

[FabianIsensee]

glad to hear it works now!