or add that

Hi AgitatedDove14 , I’m talking about the following pip install.
After that pip install, it displays agent’s conf, shows installed packages, and launches the task (no installation)
` Running in Docker mode (v19.03 and above) - using default docker image: spoter ['-e CLEARML_AGENT_SKIP_PYTHON_ENV_INSTALL=1', '-e CLEARML_AGENT_SKIP_PIP_VENV_INSTALL=1']
Running task '3ebb680b17874cda8dc7878ddf6fa735'
Storing stdout and stderr log to '/tmp/.clearml_agent_out.tsu2tddl.txt', '/tmp/.clearml_agent_o...

can you share your clearml.conf file (remove the critical information first)?

Is not direcly cached in the ~/.clearml folder. There are some directories inside (one for storage, one for pip, another for venvs, etc.

So in your case it would be stored in ~/.clearml/cache/storage_manager/datasets/ds_{ds_id}/my_file.json

mmm, can you try the following:
create a new folder with no git repo, and copy those two notebooks launch the notebook with the base task and copy the task id launch the notebook with the hyperopt task modifying the TEMPLATE_TASK_ID variable accordingly

oh 😕

each person has its laptop..and they do a pip install clearml and set their credentials in the clearml.conf file

yes, I’m talking about AWS/GCP only

ok, but except that in that case it would be clearml-server’s job to distribute to each user internally?

also I suggested to change TMPDIR env variable, since /tmp/ didn’t have a lot of space.

agent.environment.TMPDIR = ****

is it ok to see * ** * instead of the actual path?

I don’t see an agent section there 😕
Can you move your current clearml.conf file to another location and run clearml-agent init ?

the problem was docker, that had as entrypoint a bash script with python train.py --epochs=300 hardcoded, so I guess it was never acutally running the task setup from clearml.

I agree, but setting the agent’s env variable TMPDIR didn’t seem to have any effect (check the log above, it is still using /tmp )

Thanks for the answer. You’re right. I forgot to add that this tasks runs inside a docker container and I’m currently only mapping the $PWD ( ml folder) into /app folder in the container.

great! and I saw that there were some system packages needed for opencv that were installed automatically that could be turned off. Now I’m just wondering if I could remove the PIP install at the very beginning, so it starts straightaway

you can either add it manually to the installed packages, or remove the installed packages and use a setup.py file to manage the installation process

oh but he is not using docker

how do I mount my local ssh folder into /root/.ssh/ docker when running clearml-agent?
also, is there a way for it to not install the requirements, and simply run the task?

So you need to push the python files as well

this is how it looks now

it would be easier for a sysadmin to center the credentials of the bucket in the clearml-server, without the need to distribute them…every user in the server has the same credentials, and they don’t need to know them..makes sense?

Thanks TimelyPenguin76 for your answer! So indeed it was mounting it, and how do I check that “CLEARML_AGENT_SKIP_PYTHON_ENV_INSTALL” is working in my agent in docker?

I mean in the clearml-server docker

That’s why I’m suggesting him to do that 🙂

would the same experiment be called in either clearml server?

can you share you clearml.conf file? it should do that automatically if you set the development.default_output_uri key to “s3://{your_bucket}”

before the repo was already in the docker, but now it is running the agent inside the docker (so setting a virtualenv, and cloning the repo, and installing the packages)

exactly, somewhere in the docker running

right, but it’d still try to install it

oh ok, I was wondering if this could have been an issue:
agent.venvs_cache.free_space_threshold_gb = 2.0