When I Setup My Local Virtual Environment I Use A Combination Of Conda And Pip. I Use Conda As My Environment Manager, And Then Use Pip For Packages That Are Not In The Conda Repositories.

Unanswered

The error after the first iteration as follows:

` [INFO] Executing model training...

1621437621593 ecm-clearml-compute-gpu-001:0 DEBUG Epoch: 0001 TrAcc: 0.296 ValAcc: 0.005 TrPrec: 0.393 ValPrec: 0.000 TrRec: 0.296 ValRec: 0.005 TrF1: 0.262 ValF1: 0.000 TrTopK: 0.613 ValTopK: 0.026 TrLoss: 3.506 ValLoss: 5.299
Current run is terminating due to exception: cuDNN error: CUDNN_STATUS_EXECUTION_FAILED
You can try to repro this exception using the following code snippet. If that doesn't trigger the error, please include your original repro script when reporting this issue.

import torch
torch.backends.cuda.matmul.allow_tf32 = True
torch.backends.cudnn.benchmark = False
torch.backends.cudnn.deterministic = False
torch.backends.cudnn.allow_tf32 = True
data = torch.randn([16, 128, 28, 28], dtype=torch.float, device='cuda', requires_grad=True)
net = torch.nn.Conv2d(128, 128, kernel_size=[3, 3], padding=[1, 1], stride=[1, 1], dilation=[1, 1], groups=1)
net = net.cuda().float()
out = net(data)
out.backward(torch.randn_like(out))
torch.cuda.synchronize()

ConvolutionParams
data_type = CUDNN_DATA_FLOAT
padding = [1, 1, 0]
stride = [1, 1, 0]
dilation = [1, 1, 0]
groups = 1
deterministic = false
allow_tf32 = true
input: TensorDescriptor 0x564675040c30
type = CUDNN_DATA_FLOAT
nbDims = 4
dimA = 16, 128, 28, 28,
strideA = 100352, 784, 28, 1,
output: TensorDescriptor 0x564674fa4210
type = CUDNN_DATA_FLOAT
nbDims = 4
dimA = 16, 128, 28, 28,
strideA = 100352, 784, 28, 1,
weight: FilterDescriptor 0x564674fa1b60
type = CUDNN_DATA_FLOAT
tensor_format = CUDNN_TENSOR_NCHW
nbDims = 4
dimA = 128, 128, 3, 3,
Pointer addresses:
input: 0x7f151ec40000
output: 0x7f1518000000
weight: 0x7f154cd2e400
Forward algorithm: 7

Engine run is terminating due to exception: cuDNN error: CUDNN_STATUS_EXECUTION_FAILED
You can try to repro this exception using the following code snippet. If that doesn't trigger the error, please include your original repro script when reporting this issue.

Traceback (most recent call last):
File "train_clearml_pytorch_ignite_caltech_birds.py", line 104, in <module>
trainer.run()
File "/home/edmorris/.clearml/venvs-builds/3.8/lib/python3.8/site-packages/cub_tools/trainer.py", line 640, in run
self.train_engine.run(self.train_loader, max_epochs=self.config.TRAIN.NUM_EPOCHS)
File "/home/edmorris/.clearml/venvs-builds/3.8/lib/python3.8/site-packages/ignite/engine/engine.py", line 702, in run
return self._internal_run()
File "/home/edmorris/.clearml/venvs-builds/3.8/lib/python3.8/site-packages/ignite/engine/engine.py", line 775, in _internal_run
self._handle_exception(e)
File "/home/edmorris/.clearml/venvs-builds/3.8/lib/python3.8/site-packages/ignite/engine/engine.py", line 469, in _handle_exception
raise e
File "/home/edmorris/.clearml/venvs-builds/3.8/lib/python3.8/site-packages/ignite/engine/engine.py", line 745, in _internal_run
time_taken = self._run_once_on_dataset()
File "/home/edmorris/.clearml/venvs-builds/3.8/lib/python3.8/site-packages/ignite/engine/engine.py", line 850, in _run_once_on_dataset
self._handle_exception(e)
File "/home/edmorris/.clearml/venvs-builds/3.8/lib/python3.8/site-packages/ignite/engine/engine.py", line 469, in _handle_exception
raise e
File "/home/edmorris/.clearml/venvs-builds/3.8/lib/python3.8/site-packages/ignite/engine/engine.py", line 833, in _run_once_on_dataset
self.state.output = self._process_function(self, self.state.batch)
File "/home/edmorris/.clearml/venvs-builds/3.8/lib/python3.8/site-packages/cub_tools/trainer.py", line 448, in train_step
y_pred = self.model(x)
File "/home/edmorris/.clearml/venvs-builds/3.8/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
result = self.forward(*input, **kwargs)
File "/home/edmorris/.clearml/venvs-builds/3.8/lib/python3.8/site-packages/torchvision/models/resnet.py", line 249, in forward
return self._forward_impl(x)
File "/home/edmorris/.clearml/venvs-builds/3.8/lib/python3.8/site-packages/torchvision/models/resnet.py", line 238, in _forward_impl
x = self.layer2(x)
File "/home/edmorris/.clearml/venvs-builds/3.8/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
result = self.forward(*input, **kwargs)
File "/home/edmorris/.clearml/venvs-builds/3.8/lib/python3.8/site-packages/torch/nn/modules/container.py", line 119, in forward
input = module(input)
File "/home/edmorris/.clearml/venvs-builds/3.8/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
result = self.forward(*input, **kwargs)
File "/home/edmorris/.clearml/venvs-builds/3.8/lib/python3.8/site-packages/torchvision/models/resnet.py", line 74, in forward
out = self.conv2(out)
File "/home/edmorris/.clearml/venvs-builds/3.8/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
result = self.forward(*input, **kwargs)
File "/home/edmorris/.clearml/venvs-builds/3.8/lib/python3.8/site-packages/torch/nn/modules/conv.py", line 399, in forward
return self._conv_forward(input, self.weight, self.bias)
File "/home/edmorris/.clearml/venvs-builds/3.8/lib/python3.8/site-packages/torch/nn/modules/conv.py", line 395, in _conv_forward
return F.conv2d(input, weight, bias, self.stride,
RuntimeError: cuDNN error: CUDNN_STATUS_EXECUTION_FAILED
You can try to repro this exception using the following code snippet. If that doesn't trigger the error, please include your original repro script when reporting this issue.

1621437626467 ecm-clearml-compute-gpu-001:0 DEBUG Process failed, exit code 1 `

  				
Posted 
	2 years ago

					More
				  		
  Report
		
					VivaciousPenguin66
				
					0
					 × 1

98 Views

0 Answers

2 years ago

one year ago