Hi All, I Was Trying To Use Clearml-Task To Run A Custom Docker(With Poetry To Install All The Python Dependencies And Activated The Environment) Using Clearml Gpu, But It Seems Like Clearml Always Create A Virtual Environment And Run The Python Script Fr

Answered

Hi all, I was trying to use clearml-task to run a custom docker(with poetry to install all the python dependencies and activated the environment) using clearml GPU, but it seems like clearml always create a virtual environment and run the python script from /root/.clearml/venvs-builds/3.10/bin/python . Is there a way that I can have the clearml-task to automatically activated a virtual environment use the activated custom virtual environment in my docker and run the scripts from there instead of always creating a new venv inheriting from the clearml system_site_packages? I noticed that clearml.conf has a configuration agent.docker_use_activated_venv , but I am not sure how to enable it from clearml-task

  				
Posted 
	2 years ago

					More
				  		
  Report
		
					EnchantingPenguin77
				
					0
					 × 1

Votes Newest

Answers 38

@<1523701205467926528:profile|AgitatedDove14> Yes I cansee the worker:

  				
Posted 
	2 years ago

					More
				  		
  Report
		
					EnchantingPenguin77
				
					0
					 × 1

I see, like that?

  				
Posted 
	2 years ago

					More
				  		
  Report
		
					EnchantingPenguin77
				
					0
					 × 1

Click on the Task it is running and abort it, it seems to be stuck, I guess this is why the others are not pulled

  				
Posted 
	2 years ago

					More
				  		
  Report
		
					AgitatedDove14
				
					0
					 × 1

Not the file the UI

  				
Posted 
	2 years ago

					More
				  		
  Report
		
					AgitatedDove14
				
					0
					 × 1

I've added gpu:True to my hydra config file but the GPU is still not used

  				
Posted 
	2 years ago

					More
				  		
  Report
		
					EnchantingPenguin77
				
					0
					 × 1

I am using hydra in main.py

  				
Posted 
	2 years ago

					More
				  		
  Report
		
					EnchantingPenguin77
				
					0
					 × 1

well I do not think you set your pytorch lightining to use cuda:

GPU available: True (cuda), used: False
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
HPU available: False, using: 0 HPUs
/code/.venv/lib/python3.9/site-packages/lightning/pytorch/trainer/setup.py:176: PossibleUserWarning: GPU available but not used. Set `accelerator` and `devices` using `Trainer(accelerator='gpu', devices=1)`.

  				
Posted 
	2 years ago

					More
				  		
  Report
		
					AgitatedDove14
				
					0
					 × 1

I did use --args to clearml-task command for this run, but it looks like the docker didn't take it

  				
Posted 
	2 years ago

					More
				  		
  Report
		
					EnchantingPenguin77
				
					0
					 × 1

Show more results

Write your answer

143K Views

38 Answers

2 years ago