I used an env variable to avoid creating and endless loop of init/enqueue (using an argument like clearml.queue that would be captured and forwarded to the agent)
I tried like that clearml-task --script train.py --args overrides="log.clearml=True train.epochs=200 clearml.save=True" --project mornag-plain-dense --name mornag-plain-dense-training --queue tesla-t4 --skip-task-init
I see TrickyFox41 try the following:--args overrides="param=value"
Notice this will change the Args/overrides argument that will be parsed by hydra to override it's params
same result. in print(cfg) there are no overrides
Hi TrickyFox41
Hey since Hydra does not work with
clearml-task
I should shouldn't it? what does not work ?
Yes the task is running on a remote agent with the --docker
flag
this is the config on the machine(s) running the agent
`
agent {
venvs_cache: {
max_entries: 50
free_space_threshold_gb: -1
path: ~/.clearml/venvs-cache
}
extra_docker_arguments: [
"--network", "host",
"-v", "/home/ubuntu/.ssh:/root/.ssh:ro",
"-v", "/home/ubuntu/.cache:/root/.cache",
]
docker_internal_mounts {
sdk_cache: "/clearml_agent_cache"
apt_cache: "/var/cache/apt/archives"
ssh_folder: "~/.ssh"
ssh_ro_folder: "/.ssh"
pip_cache: "/root/.pip-cache"
poetry_cache: "/root/.pypoetry-cache"
vcs_cache: "/root/.clearml/vcs-cache"
venv_build: "~/.clearml/venvs-builds"
pip_download: "/root/.clearml/pip-download-cache"
}
}
sdk {
aws {
s3 {
host: "...:443"
key: "..."
secret: "..."
secure: true
}
}
}
api {
web_server: https://...
api_server: https://...
file_server: https://...
credentials {
}
} `
i printed cfg
in the script and the config has not been overwritten 😢
Hmm that should have worked ...
I'm assuming the Task itself is running on a remote agent, correct ?
Can you see the changes in the OmegaConf section ?
what happens when you pass--args overrides="['dataset.path=abcd']"
This is what it looks like from the web app
Hmm can you try:--args overrides="['log.clearml=True','train.epochs=200','clearml.save=True']"
--args param.ovveride=value
does not work with clearml-task
TrickyFox41 are you saying that if you add Task.init inthe code it works, but when you are calling "clearml-task" it does not work? (in both cases editing the Args/overrides ?
No luck with --args overrides="['dataset.path=abcd']"
By the way, since if i create the task locally, reset it and enqueue it, it works. This is the workaround that i'm using right now
What do you have under Hydra section? and OmegaConf section
Related GitHub issue https://github.com/allegroai/clearml/issues/847
So I think there are two bugs here?
--args overrides="key=value" does not work request: add --hydra to override hydra arguments (and if this is added the first one is not needed)Is that correct?