more than 120s?

Maybe there is setting in docker to move the space used in a different location? I can simply increase the storage of the first disk, no problem with that

I am doing:
try: score = get_score_for_task(subtask) except: score = pd.NA finally: df_scores = df_scores.append(dict(task=subtask.id, score=score, ignore_index=True) task.upload_artifact("metric_summary", df_scores)

yes, because it won’t install the local package which has this setup.py with the problem in its install_requires described in my previous message

that works from within the ssh session

SuccessfulKoala55

In the docker-compose file, you have an environment setting for the apiserver service host and port (CLEARML_ELASTIC_SERVICE_HOST and CLEARML_ELASTIC_SERVICE_PORT) - changing those will allow you to point the server to another ES service

The ES cluster is running in another machine, how can I set its IP in CLEARML_ELASTIC_SERVICE_HOST ? I would need to add host to the networks of the apiserver service somehow? How can I do that?

AgitatedDove14 any chance you found something interesting? 🙂

I am trying to upload an artifact during the execution

I was asking to exclude this possibility from my debugging journey 😁

https://github.com/allegroai/clearml/issues/547

I will try adding
sudo sh -c "echo '\n* soft nofile 65535\n* hard nofile 65535' >> /etc/security/limits.conf"to the extra_vm_bash_script , maybe that’s enough actually

venv mode

Awesome! Thanks! 🙏

yes 🙂

AgitatedDove14 The first time it installs and create the cache for the env, the second time it fails with:
Applying uncommitted changes ERROR: Directory '.' is not installable. Neither 'setup.py' nor 'pyproject.toml' found. clearml_agent: ERROR: Command '['/home/user/.clearml/venvs-builds.1/3.6/bin/python', '-m', 'pip', '--disable-pip-version-check', 'install', '-r', '/tmp/cached-reqsmncaxx45.txt']' returned non-zero exit status 1.

by replacing the pid with $PID ?

Yes, I get 0 afterwards

Should I try to disable dynamic mapping before doing the reindex operation?

Will the from clearml import Task raise an error if no clearml.conf exists? Or only when actual features requiring to define the server (such as Task.init ) will be called

Yes, but I am not certain how: I just deleted the /data folder and restarted the server

nvm, bug might be from my side. I will open an issue if I find any easy reproducible example

The number of documents in the old and the new env are the same though 🤔 I really don’t understand where this extra space used comes from

PS: in the new env, I’v set num_replicas: 0, so I’m only talking about primary shards…

The only thing that changed is the new auth.fixed_users.pass_hashed field, that I don’t have in my config file

Nice, thanks!

Hi AgitatedDove14 , coming by after a few experiments this morning:
Indeed torch 1.3.1 does not support cuda, I tried with 1.7.0 and it worked, BUT trains was not able to pick the right wheel when I updated the torch req from 1.3.1 to 1.7.0: It downloaded wheel for cuda version 101. But in the experiment log, the agent correctly reported the cuda version (111). I then replaced the torch==1.7.0 with the direct https link to the torch wheel for cuda 110, and that worked (I also tried specifyin...

Nice, seems to work! 🎉

Note: I can verify that post_packages is well picked up by the trains-agent, since in the experiment log I see:
agent.package_manager.type = pip agent.package_manager.pip_version = \=\=20.2.3 agent.package_manager.system_site_packages = true agent.package_manager.force_upgrade = false agent.package_manager.post_packages.0 = PyJWT\=\=1.7.1

No space at the end of the diff file:
` diff --git a/configs/2.2.2_from_scratch.yaml b/configs/2.2.2_from_scratch.yaml
index 9fece48..5816f78 100644
--- a/configs/2.2.2_from_scratch.yaml
+++ b/configs/2.2.2_from_scratch.yaml
@@ -136,7 +136,7 @@ data_processing:
optimizer:
type: 'RMSprop'
args:

• lr: 2.5e-4

• lr: 1.5e-5
  momentum: 0
  weight_decay: 0 `

Sure, where can I find this file?