oh but he is not using docker

ClearML downloads/caches datasets to ~/.clearml/ folder so yes, you need to modify your code.
dataset_folder = Dataset.get(project_name=, dataset_name=, version=).get_local_copy() file_json_path = os.path.join(dataset_folder, 'file.json')

ok, but if you were to run it from a different machine (or a different user!) it wouldn’t work

you would, but I’d advise against it, since that is not the intended way

I’m suggesting MagnificentWorm7 to do that yes, instead of adding the files to a ClearML dataset in each step

I could map the root folder of the repo into the container, but that would mean everything ends up in there

but it would only be affecting that session in the terminal…so you would want to add it to your .bashrc

thanks!

exactly, somewhere in the docker running

oh 😕

so when inside the docker, I don’t see the git repo and that’s why ClearML doesn’t see it

I’m afaid I don’t think there is a way to go around this without modifying your code.

To give more context, he is running an hyper params optimization script, that internally clones a base task and runs it with certain params and checks if a metric increases or decreases. It is when the agent tries to run this task that the error raises.

ERROR: Could not install packages due to an EnvironmentError: [Errno 28] No space left on device
clearml_agent: ERROR: Could not install task requirements!
Command '['~/.clearml/venvs-builds/3.8/bin/python', '-m', 'pip', '--disable-pip-v...

any idea what could be the issue @<1523701087100473344:profile|SuccessfulKoala55> ?

also I suggested to change TMPDIR env variable, since /tmp/ didn’t have a lot of space.

agent.environment.TMPDIR = ****

is it ok to see * ** * instead of the actual path?

ok, but except that in that case it would be clearml-server’s job to distribute to each user internally?

Hi AgitatedDove14 , I’m talking about the following pip install.
After that pip install, it displays agent’s conf, shows installed packages, and launches the task (no installation)
` Running in Docker mode (v19.03 and above) - using default docker image: spoter ['-e CLEARML_AGENT_SKIP_PYTHON_ENV_INSTALL=1', '-e CLEARML_AGENT_SKIP_PIP_VENV_INSTALL=1']
Running task '3ebb680b17874cda8dc7878ddf6fa735'
Storing stdout and stderr log to '/tmp/.clearml_agent_out.tsu2tddl.txt', '/tmp/.clearml_agent_o...

line 120 says unmark to enable venv caching (it comes commented by default, but since I’m copying my conf it isn’t commented there)

Hi Matias! 🙌

how do I mount my local ssh folder into /root/.ssh/ docker when running clearml-agent?
also, is there a way for it to not install the requirements, and simply run the task?

in linux you can run in a terminal:
export CLEARML_CONFIG_FILE=/new/path/to/conf

would it be possible to change de dataset.add_files to some function that moves your files to a common folder (local or cloud), and then use the last step in the dag to create the dataset using that folder?

Thanks TimelyPenguin76 for your answer! So indeed it was mounting it, and how do I check that “CLEARML_AGENT_SKIP_PYTHON_ENV_INSTALL” is working in my agent in docker?

each person has its laptop..and they do a pip install clearml and set their credentials in the clearml.conf file

this is how it looks now

where is the dataset stored? maybe you deleted the credentials by mistake? or maybe you are not installing the libraries needed (for example if using AWS you need boto3, if GCP you need google-cloud-storage)

I agree, but setting the agent’s env variable TMPDIR didn’t seem to have any effect (check the log above, it is still using /tmp )

There’s only a jupyter notebook

just do:
import os.path as op dataset_folder = Dataset.get(dataset_id="...").get_local_copy() csv_file = op.join(dataset_folder, 'salary.csv')