Reputation
Badges 1
6 × Eureka!Latest clearml
version I believe (1.4.0)
Hi SweetBadger76 , I have not been able to deal with this issue yet. I am getting all sorts of weird behaviours which are likely due to some miss configuration of my ClearML agents or of the experiments I am trying to run. The latest one is that ClearML agents are ignoring my --docker
flag and running everything on the host machine using an env. On this, can you clarify something for me: if I clone an experiment, will the configs on the experiment overwrite the ones from the agent? For...
TimelyPenguin76 Thanks for you suggestion! I’ve considered using that but, from what I understood, it doesn’t offer the same functionalities.
For example, by creating the task within the script it already identifies the branch/commit being used and also includes uncommitted changes. I also have auxiliary functions to guarantee that the experiments go to the correct projects within ClearML according to the script that is being used. It would also require a significant change to the command li...
TimelyPenguin76 SweetBadger76 thanks for the support!
I ran the script on the terminal (powershell) using a command similar to python -m <path>.<to>.<module> --arg1 <arg1_value> --arg2 <arg2_value> ...
I ran it on Windows. The ClearML server is running on Ubuntu.
I will create a minimal program that reproduces the error and come back to you (I will also test both on WSL and Ubuntu to have a better idea if it is OS specific)
Update on this one, I noticed I had different versions of clearml
in my dev machine and the training machine (host and container). Updating both to the latest 1.4.1
caused a different error (related to the other question I posted in the channel) where it tries to install the packages from my dev machine (windows) in the docker container used in the training machine (ubuntu container, ubuntu host). The main issue I am trying to get around now is that I use pycocotools
which ha...
SuccessfulKoala55 thanks for the reply! I think I have a very specific use case.
I am calling execute_remotely
from a Windows machine which has pycocotools-windows
installed, but my docker container is running Ubuntu with pycocotools
installed, and even with CLEARML_AGENT_SKIP_PIP_VENV_INSTALL
set it tries to install the windows specific package and fails.
Is there a way to completely skip verifying the requirements?