RuntimeError: cuDNN error: CUDNN_STATUS_NOT_INITIALIZED #759

johnny12150 · 2021-03-07T07:14:32Z

johnny12150
Mar 7, 2021

I have tested Caser, NARM, and 'SASRecwith datasets astmall, 'diginetica and yoochoose in version 0.2.0.
They all encounter the same error as below.

  File "/home/wade/.local/lib/python3.7/site-packages/recbole/trainer/trainer.py", line 262, in fit
    train_loss = self._train_epoch(train_data, epoch_idx, show_progress=show_progress)
  File "/home/wade/.local/lib/python3.7/site-packages/recbole/trainer/trainer.py", line 152, in _train_epoch
    losses = loss_func(interaction)
  File "/home/wade/.local/lib/python3.7/site-packages/recbole/model/sequential_recommender/caser.py", line 144, in calculate_loss
    seq_output = self.forward(user, item_seq)
  File "/home/wade/.local/lib/python3.7/site-packages/recbole/model/sequential_recommender/caser.py", line 108, in forward
    out_v = self.conv_v(item_seq_emb)
  File "/home/wade/.local/lib/python3.7/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/wade/.local/lib/python3.7/site-packages/torch/nn/modules/conv.py", line 399, in forward
    return self._conv_forward(input, self.weight, self.bias)
  File "/home/wade/.local/lib/python3.7/site-packages/torch/nn/modules/conv.py", line 396, in _conv_forward
    self.padding, self.dilation, self.groups)
RuntimeError: cuDNN error: CUDNN_STATUS_NOT_INITIALIZED

These models were all runnable in the previous version.
I first thought this error may cause by OOM and yet they ended before taking 20% off VRAM.
I have only tested it on 1 Linux server with two 2070S.
I probably will test these models on other PCs.

Answered by johnny12150

Mar 10, 2021

After I found the error source on Windows, I will open a bug issue.

View full answer

EliverQ · 2021-03-07T08:29:21Z

EliverQ
Mar 7, 2021
Maintainer

Hi, @johnny12150 ! The three datasets all have different sub-datasets, so does this problem emerge whenever you use any sub-datasets? And could you please provide your .yaml file for me?

3 replies

johnny12150 Mar 7, 2021
Author

Sure. I will run the test also on Windows then gather the error info and sub-datasets info.
Config

USER_ID_FIELD: user_id
load_col:
  inter: [user_id, item_id, timestamp]
epochs: 30
train_batch_size: 100
eval_batch_size: 100  # val and test batch_size
topk: [10, 20]
valid_metric: MRR@20
stopping_step: 5  # early stop after num epochs
split_ratio: [0.8,0.1,0.1]

johnny12150 Mar 7, 2021
Author

By the way, can I set the hyper-parameter in yaml file?

hyp1231 Mar 8, 2021
Maintainer

Yes, hyper-parameters can be set in yaml files.

johnny12150 · 2021-03-07T09:51:30Z

johnny12150
Mar 7, 2021
Author

For tmall dataset (not_merged version), I tried both buy and click download from google drive.
The NARM error would be the following:

Trainable parameters: 140941696
Train     0:   0%|          | 0/670518 [00:00<?, ?it/s]
Traceback (most recent call last):
  File "F:/volume/SR baseline/run_recbole.py", line 24, in <module>
    run_recbole(model=args.model, dataset=args.dataset, config_file_list=config_file_list, config_dict=arg_dict)
  File "C:\Users\Wade\Anaconda3\envs\rosetta_GNN\lib\site-packages\recbole\quick_start\quick_start.py", line 54, in run_recbole
    train_data, valid_data, saved=saved, show_progress=config['show_progress']
  File "C:\Users\Wade\Anaconda3\envs\rosetta_GNN\lib\site-packages\recbole\trainer\trainer.py", line 262, in fit
    train_loss = self._train_epoch(train_data, epoch_idx, show_progress=show_progress)
  File "C:\Users\Wade\Anaconda3\envs\rosetta_GNN\lib\site-packages\recbole\trainer\trainer.py", line 152, in _train_epoch
    losses = loss_func(interaction)
  File "C:\Users\Wade\Anaconda3\envs\rosetta_GNN\lib\site-packages\recbole\model\sequential_recommender\narm.py", line 99, in calculate_loss
    seq_output = self.forward(item_seq, item_seq_len)
  File "C:\Users\Wade\Anaconda3\envs\rosetta_GNN\lib\site-packages\recbole\model\sequential_recommender\narm.py", line 82, in forward
    c_global = ht = self.gather_indexes(gru_out, item_seq_len - 1)
  File "C:\Users\Wade\Anaconda3\envs\rosetta_GNN\lib\site-packages\recbole\model\abstract_recommender.py", line 119, in gather_indexes
    output_tensor = output.gather(dim=1, index=gather_index)
RuntimeError: gather_out_cpu(): Expected dtype int64 for index

The error of Caser would be:

Trainable parameters: 77608632
Train     0:   1%|██▏                                                                                                                                                              | 1190/88942 [20:55<25:42:41,  1.05s/it]
Traceback (most recent call last):
  File "run_recbole.py", line 24, in <module>
    run_recbole(model=args.model, dataset=args.dataset, config_file_list=config_file_list, config_dict=arg_dict)
  File "C:\Users\Wade\Anaconda3\lib\site-packages\recbole\quick_start\quick_start.py", line 54, in run_recbole
    train_data, valid_data, saved=saved, show_progress=config['show_progress']
  File "C:\Users\Wade\Anaconda3\lib\site-packages\recbole\trainer\trainer.py", line 262, in fit
    train_loss = self._train_epoch(train_data, epoch_idx, show_progress=show_progress)
  File "C:\Users\Wade\Anaconda3\lib\site-packages\recbole\trainer\trainer.py", line 160, in _train_epoch
    self._check_nan(loss)
  File "C:\Users\Wade\Anaconda3\lib\site-packages\recbole\trainer\trainer.py", line 227, in _check_nan
    raise ValueError('Training loss is nan')
ValueError: Training loss is nan

And the error seems to be the same one as NARM.

Trainable parameters: 140921984
Train     0:   0%|          | 0/670518 [00:00<?, ?it/s]
Traceback (most recent call last):
  File "F:/volume/SR baseline/run_recbole.py", line 24, in <module>
    run_recbole(model=args.model, dataset=args.dataset, config_file_list=config_file_list, config_dict=arg_dict)
  File "C:\Users\Wade\Anaconda3\envs\rosetta_GNN\lib\site-packages\recbole\quick_start\quick_start.py", line 54, in run_recbole
    train_data, valid_data, saved=saved, show_progress=config['show_progress']
  File "C:\Users\Wade\Anaconda3\envs\rosetta_GNN\lib\site-packages\recbole\trainer\trainer.py", line 262, in fit
    train_loss = self._train_epoch(train_data, epoch_idx, show_progress=show_progress)
  File "C:\Users\Wade\Anaconda3\envs\rosetta_GNN\lib\site-packages\recbole\trainer\trainer.py", line 152, in _train_epoch
    losses = loss_func(interaction)
  File "C:\Users\Wade\Anaconda3\envs\rosetta_GNN\lib\site-packages\recbole\model\sequential_recommender\sasrec.py", line 127, in calculate_loss
    seq_output = self.forward(item_seq, item_seq_len)
  File "C:\Users\Wade\Anaconda3\envs\rosetta_GNN\lib\site-packages\recbole\model\sequential_recommender\sasrec.py", line 121, in forward
    output = self.gather_indexes(output, item_seq_len - 1)
  File "C:\Users\Wade\Anaconda3\envs\rosetta_GNN\lib\site-packages\recbole\model\abstract_recommender.py", line 119, in gather_indexes
    output_tensor = output.gather(dim=1, index=gather_index)
RuntimeError: gather_out_cpu(): Expected dtype int64 for index

Process finished with exit code 1

2 replies

johnny12150 Mar 8, 2021
Author

loss nan happened even when I used the previous version.

johnny12150 Mar 8, 2021
Author

However, NARM and SASRec indeed work in the previous version among the three datasets.

johnny12150 · 2021-03-07T14:00:47Z

johnny12150
Mar 7, 2021
Author

I have tested the 0.1.2 version with PyTorch 1.7 & 1.8 and these three models work properly.

0 replies

hyp1231 · 2021-03-08T10:29:35Z

hyp1231
Mar 8, 2021
Maintainer

Could you please provide versions of:

python
pytorch
cudatoolkit

so that we can test on it, thanks.

1 reply

johnny12150 Mar 8, 2021
Author

python 3.5 & 3.7 & 3.8
pytorch 1.6, 1.7.1, 1.8
cudatoolkit 10.2 & 11.0
These are the ones that I have personally tested.

EliverQ · 2021-03-08T13:18:26Z

EliverQ
Mar 8, 2021
Maintainer

We've tested on below versions with your yaml file but it works:

python 3.7
pytorch 1.6
cudatoolkit 11.0
So maybe there are some bugs when runing on Windows? We are going to test that.

1 reply

johnny12150 Mar 8, 2021
Author

It only happened on version 0.2.0 of recbole and it was tested on Linux, so it may not be a Windows issue. This machine use cuda10.2 and with pytorch1.8.0 and python 3.8. I have tested on 2 Windows and 2 Linux they acted the same.

EliverQ · 2021-03-10T03:39:18Z

EliverQ
Mar 10, 2021
Maintainer

Hi, @johnny12150 . We've tested these three models with your environment but it still works. Would you please check your environment again?

1 reply

johnny12150 Mar 10, 2021
Author

What kind of environments you have tested with?

johnny12150 · 2021-03-10T03:43:44Z

johnny12150
Mar 10, 2021
Author

Below is the complete list that I have tested with and the three models all failed.

Win10, python3.8, cuda11, pytorch1.7.1, recbole0.2.0
Win10, python3.8, cuda11, pytorch1.8.0, recbole0.2.0
Win10, python3.7, cuda10.2, pytorch1.7.1, recbole0.2.0
Linux, python3.7.4, cuda10.2, pytorch1.7.1, recbole0.2.0
Linux, python3.7.4, cuda10.2, pytorch1.8.0, recbole0.2.0
Personally, I encounter some unknown errors on PyTorch 1.8.0, so I downgrade it.

The reason why ItemKNN stuck is still unknown to me.

3 replies

johnny12150 Mar 10, 2021
Author

Maybe I need to take a while to trace the type error for the index.
File "C:\Users\Wade\Anaconda3\envs\rosetta_GNN\lib\site-packages\recbole\model\abstract_recommender.py", line 119, in gather_indexes output_tensor = output.gather(dim=1, index=gather_index)
Yet, I still don't have any clue why the training loss is nan.

EliverQ Mar 10, 2021
Maintainer

Below is the complete list that I have tested with and the three models all failed.

Win10, python3.8, cuda11, pytorch1.7.1, recbole0.2.0

Win10, python3.8, cuda11, pytorch1.8.0, recbole0.2.0

Win10, python3.7, cuda10.2, pytorch1.7.1, recbole0.2.0

Linux, python3.7.4, cuda10.2, pytorch1.7.1, recbole0.2.0

Linux, python3.7.4, cuda10.2, pytorch1.8.0, recbole0.2.0
Personally, I encounter some unknown errors on PyTorch 1.8.0, so I downgrade it.

The reason why ItemKNN stuck is still unknown to me.

We've tested in Linux, python3.7/3.8, cuda10.2, pytorch1.8.0, recbole0.2.0. And ItemKNN stuck problem actually exists but it hasn't been located.

johnny12150 Mar 10, 2021
Author

Ok, it seems this may be a Windows issue.
(I notice those models work great with 0.1.2 on Windows)
I will check that if I can somehow run on my Linux machines.

johnny12150 · 2021-03-10T06:13:47Z

johnny12150
Mar 10, 2021
Author

Yap, the RuntimeError: gather_out_cuda(): Expected dtype int64 for index issue seems only happened on Windows or the higher version of pytorch. #707

2 replies

johnny12150 Mar 10, 2021
Author

I checked on two Linux servers.
If I use PyTorch 1.7/ 1.7 with cu101 and recbole 0.2.0 installed with pip, it works properly.
However, when I use conda to install pytorch, it just somehow failed.
I still don't have a fix or any suitable combination for Windows to work with recbole 0.2.0.
Will you test on Windows and try to fix the issue?

EliverQ Mar 10, 2021
Maintainer

Yap, the RuntimeError: gather_out_cuda(): Expected dtype int64 for index issue seems only happened on Windows or the higher version of pytorch. #707

Sorry for this bug, but #707 has been fixed in #710. We'll try to locate and fix the bug as soon as possible.

johnny12150 · 2021-03-10T06:44:40Z

johnny12150
Mar 10, 2021
Author

After I found the error source on Windows, I will open a bug issue.

1 reply

EliverQ Mar 10, 2021
Maintainer

Ok, thank you for your help!

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

RuntimeError: cuDNN error: CUDNN_STATUS_NOT_INITIALIZED #759

{{title}}

Replies: 9 comments 14 replies

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

Select a reply

RuntimeError: cuDNN error: CUDNN_STATUS_NOT_INITIALIZED #759

johnny12150 Mar 7, 2021

Replies: 9 comments · 14 replies

EliverQ Mar 7, 2021 Maintainer

johnny12150 Mar 7, 2021 Author

johnny12150 Mar 7, 2021 Author

hyp1231 Mar 8, 2021 Maintainer

johnny12150 Mar 7, 2021 Author

johnny12150 Mar 8, 2021 Author

johnny12150 Mar 8, 2021 Author

johnny12150 Mar 7, 2021 Author

hyp1231 Mar 8, 2021 Maintainer

johnny12150 Mar 8, 2021 Author

EliverQ Mar 8, 2021 Maintainer

johnny12150 Mar 8, 2021 Author

EliverQ Mar 10, 2021 Maintainer

johnny12150 Mar 10, 2021 Author

johnny12150 Mar 10, 2021 Author

johnny12150 Mar 10, 2021 Author

EliverQ Mar 10, 2021 Maintainer

johnny12150 Mar 10, 2021 Author

johnny12150 Mar 10, 2021 Author

johnny12150 Mar 10, 2021 Author

EliverQ Mar 10, 2021 Maintainer

johnny12150 Mar 10, 2021 Author

EliverQ Mar 10, 2021 Maintainer

johnny12150
Mar 7, 2021

Replies: 9 comments 14 replies

EliverQ
Mar 7, 2021
Maintainer

johnny12150 Mar 7, 2021
Author

johnny12150 Mar 7, 2021
Author

hyp1231 Mar 8, 2021
Maintainer

johnny12150
Mar 7, 2021
Author

johnny12150 Mar 8, 2021
Author

johnny12150 Mar 8, 2021
Author

johnny12150
Mar 7, 2021
Author

hyp1231
Mar 8, 2021
Maintainer

johnny12150 Mar 8, 2021
Author

EliverQ
Mar 8, 2021
Maintainer

johnny12150 Mar 8, 2021
Author

EliverQ
Mar 10, 2021
Maintainer

johnny12150 Mar 10, 2021
Author

johnny12150
Mar 10, 2021
Author

johnny12150 Mar 10, 2021
Author

EliverQ Mar 10, 2021
Maintainer

johnny12150 Mar 10, 2021
Author

johnny12150
Mar 10, 2021
Author

johnny12150 Mar 10, 2021
Author

EliverQ Mar 10, 2021
Maintainer

johnny12150
Mar 10, 2021
Author

EliverQ Mar 10, 2021
Maintainer