Yes, so here you have the three task (here is a slight refactor using task_func instead of task but the result is the same)

1- all different (status pending)
2- two equal (those which started)
3- all equal (all running or completed)

im running them with python my_script.py -m my_parameter=value_1,value_2,value_3 (using hydra multirun)

each of those runs finished producing 10 plots each but in clearml only 1, a few, or none got uploaded

multirun is not working as expected
when I run python run.py -m env=gpu clearml.task_name=demo_all_models "model=glob(*)"
it should run remotely one run per model
this is the output I see locally
` ╰─ python run.py -m env=gpu clearml.task_name=demo_all_models "model=glob(*)"
2022/09/13 20:49:31 WARNING mlflow.utils.autologging_utils: You are using an unsupported version of pytorch. If you encounter errors during autologging, try upgrading / downgrading pytorch to a supported version, or...

it doesnt happen with all the tasks of the multirun as you can see in the photo

yes

dont think will be reproducible with the hydra example. It was just that I launched like 50 jobs and some of them because of the parameters maybe failed (strangely with no error).
But is ok for now I guess, will debug wether those experiments that failed would failed if ran independently as well

indeed, im looking at their corresponding multirun outputs folder and the logs terminate before without error and the only plots saved are those in clearml. So as you say, it seems hydra kills these

that did it! 🙌 thank you!

still the same result. What's strange is that the remote jobs, as soon as they are launched, if I compare their configs while in state pending, they have the right all different configs, but later, while running, they all revent to the same config by the end

` ─ python run.py -m env=gpu clearml.task_name=connect_test "model=glob(*)" trainer_params.max_epochs=5
2022/09/14 01:10:07 WARNING mlflow.utils.autologging_utils: You are using an unsupported version of pytorch. If you encounter errors during autologging, try upgrading / downgrading pytorch to a supported version, or try upgrading MLflow.
/Users/juan/mindfoundry/git_projects/cvae/run.py:38: UserWarning:
The version_base parameter is not specified.
Please specify a compatability version level...

yes!

actually I really need help with this, ive been struggling for 2 days to make the aws autoscaler work.
what I want:
do a multirun with hydra where each of the runs get executed remotely

my implementation (iterated over several using create_function_task
, etc:

` @hydra.main(config_path="configs", config_name="ou_cvae")
def main(config: DictConfig):
curr_dir = Path(file).parent
if config.clearml.enabled:
# Task.force_requirements_env_freeze(requirements_file=str(cur...

using 1.3.0

Im using the latest version of clearml and clearml-agenst and im seeing the same error

I guess one solution would be to write a clearml https://hydra.cc/docs/advanced/plugins/overview/ for hydra, like the one with ray.
I leave it here though for now (end of POC)

I see, thank you!

is your server. Will check that example later

yes

it also happens with other configuration values like this one which is a boolean. I think it happens in general with configuration values that are passed in your run command as flags (using the override syntax of hydra)

is an integer when it exists

waiting now for the run...

but I still have the problem if I try to run locally for debugging purposes clearml-agent execute --id ...

yes, the remote task is working 🙂

I find this error if I try to run any of the runs generated
clearml_agent: ERROR: Could not find task id=a270d2a56feb475181ef3c9c82111b7f (for host: some_secret_host) Exception: __init__() got an unexpected keyword argument 'types'

my bad :man-facepalming: the hydra error is because the data config folder is not commited (gitignore)

ok, yes, but this will install the package of the branch specified there.
So If im working on my own branch and want to run an experiment, I would have to manually put in the git path my current branch name. I guess I can add some logic to get the current branch from the env. Thank you

yes! thanks

found the env freeze. For the second workflow all I would need I guess then would be and env variable that would tell me whether this is being currently run by an agent or not

basically running_locally() ok, I think I have everything I need. Will give it a try.

im trying to use https://clear.ml/docs/latest/docs/webapp/applications/apps_aws_autoscaler .
In the setup, I have to provide a personal access token (PAC) from git.
The agents when setting up the env to run the tasks from the queue cannot clone the repo using the pac

cloning: git@gitlab.com:<redacted>.git Using user/pass credentials - replacing ssh url 'git@gitlab.com:<redacted>.git' with https url ' ` <redacted>.git'
Host key verification failed.
fatal: Could not read from remote repos...