Hi. I Get Some Problem With Clearml Agent. I Start Training On My Local Device, Clone Run, And Start This Run In Docker On Cluster. But, Seems Like Clearml Agent Сaches Environment(Package Weels, Python Version, Etc). Can I Config Clearml Agent To Not Сac

Unanswered

Hi StickyBlackbird93
Yes, this agent version is rather old ( clearml_agent v1.0.0 )
it had a bug where pytorch wheel aaarch broke the agent (by default the agent in docker mode, will use the latest stable version, but not in venv mode)
Basically upgrade to the latest clearml-agent version it should solve the issue:
pip3 install -U clearml-agemnt==1.2.3BTW for future debugging, this is the interesting part of the log (Notice it is looking for the correct pytorch based on the auto detected cuda version, 11.5)
Then it failed because it found aarch instead of x86 (this is the bug that was fixed in the latest version)
1654011488836 sjc13-t04-mlt02:!6e9:gpu0 DEBUG Torch CUDA 115 download page found Found PyTorch version torch==1.11.0 matching CUDA version 115 [31mERROR: torch-1.11.0-cp39-cp39-manylinux2014_aarch64.whl is not a supported wheel on this platform.[0m[31m

  				
Posted 
	one year ago

					More
				  		
  Report
		
					AgitatedDove14
				
					0
					 × 1

106 Views

0 Answers

one year ago