ClearML FAQ | Hi Everyone, Is There A Way To Avoid The Environment Setup When Running A Task Using A Worker? I Am Currently Using A Custom Docker Image That Already Has All The Require Packages Installed. I Tried Setting The Env Var

Answered

Hi Everyone, Is There A Way To Avoid The Environment Setup When Running A Task Using A Worker? I Am Currently Using A Custom Docker Image That Already Has All The Require Packages Installed. I Tried Setting The Env Var

Hi everyone, is there a way to avoid the environment setup when running a task using a worker?
I am currently using a custom docker image that already has all the require packages installed. I tried setting the env var CLEARML_AGENT_SKIP_PIP_VENV_INSTALL=/usr/bin/python both for the docker container and for host machine (one at a time and at the same time) but it still tries to install everything

  				
Posted 
	2 years ago

					More
				  		
  Report
		
					SteepDeer88
				
					0
					 × 1

Votes Newest

Answers 3

SteepDeer88
Try the following:
` Task.add_requirements("pycocotools-windows", "; platform_system == "Windows"")
Task.add_requirements("pycocotools", "; platform_system != "Windows"")

Task.init(...) You should see in your "installed packages" something like: pycocotools-windows ; platform_system == "Windows"
pycocotools ; platform_system != "Windows" `

  				
Posted 
	2 years ago

					More
				  		
  Report
		
					AgitatedDove14
				
					0
					 × 1

SuccessfulKoala55 thanks for the reply! I think I have a very specific use case.
I am calling execute_remotely from a Windows machine which has pycocotools-windows installed, but my docker container is running Ubuntu with pycocotools installed, and even with CLEARML_AGENT_SKIP_PIP_VENV_INSTALL set it tries to install the windows specific package and fails.
Is there a way to completely skip verifying the requirements?

  				
Posted 
	2 years ago

					More
				  		
  Report
		
					SteepDeer88
				
					0
					 × 1

Hi SteepDeer88 , if you use CLEARML_AGENT_SKIP_PIP_VENV_INSTALL the agent will not install the virtual environment. It will still verify the requests, but if you already have these installed, it should not take any time

  				
Posted 
	2 years ago

					More
				  		
  Report
		
					SuccessfulKoala55
				
					0
					 × 1

Write your answer

1K Views

3 Answers

2 years ago

one year ago

Tags

Similar posts

Hi Everyone, I'M Trying To Deploy My First Clearml Pipeline With A Configuration For Logging And Caching To Avoid Repeating Already Computed Steps. However, The Caching Doesn'T Seem To Be Working Correctly. Despite Not Changing The Configuration, The Firs

Hi, When Running A Task, Is There A Way To Run A Script Already In The Container? For Example There Is A Script /Home/Root/Entrypoint.Sh I Would Like To Avoid Specifying A Repo Or A Local Script On My Machine.

Hi Again, I Am Trying To Execute A Pipeline Remotely, However I Am Running Into A Problem With The Steps That Require A Local Package. Basically I Have A Repo, That I Created Specifically For This Pipeline And I Have Packaged It So That I Can Split It I

Hey Folks, I Noticed A Weird Behaviour (I Think) Where The Reported Histogram Plots For Different Iterations Of The Experiment Vanished And Only Remained Visible On The Last Iteration. Is This An Expected Event? Or Should I Be Adding A Parameter To Avoid

Hi, Can We Specify The Limit And Request Resource Differently For The Workers Spawned By The Clearml Agent? So Some Tasks May Require A Lot Of Ram, But Other Tasks Don'T. So I Don'T Them To All Request The Same Amount Of Resource To Run. Edit: My Clearml

Started Using The Integrated Gcp Autoscaler To Avoid Some Problems We Had. For Some Reason The Instances Doesn'T Have A Gpu Although Specifically Defined In The Ui. How Come? (Not Using Any Docker Container For The Agents)

Hi, I Have Clearml Installed Locally And I Have A Question Regarding Saving Of Experiments And Models. In A Project If We Do Not Change The Task Name Does It Override The Previous Experiment With Same Name And How To Avoid It Without Changing The Task Nam

Hello, I’Ve Enabled The Web Authentication For My Clearml Server And I’M Wondering Whether There’S A Way To Avoid Restarting All The Clearml Services Every Time A New User Is Added. For Example If Some Experiments Are Ongoing A New User Can’T Be Added Wit

Hi There, I Have A Bit Of A Problem With Aws Secrets: I Pass Keys As Env Var To Clearml-Agents To Retrieve Data From A Bucket In Us-East-1 But I Use A Bucket To Store Task Artifacts In A Bucket In Eu-Central-1. So When I Pass Aws Keys As Env Vars, The Tas

Hi, When Executing Remotely, Is It Possible To Provide A Requirements.Txt? I Am Sending My Script From A Windows Pc To A Linux Machine So Some Of The Sys Related Libraries Are Not Required And So I Wanted To Avoid The Automatic "Force_Requirements_Env_Fre