I see TrickyFox41 try the following:--args overrides="param=value"
Notice this will change the Args/overrides argument that will be parsed by hydra to override it's params
Related GitHub issue https://github.com/allegroai/clearml/issues/847
By the way, since if i create the task locally, reset it and enqueue it, it works. This is the workaround that i'm using right now
i printed cfg
in the script and the config has not been overwritten 😢
Hi TrickyFox41
Hey since Hydra does not work with
clearml-task
I should shouldn't it? what does not work ?
So I think there are two bugs here?
--args overrides="key=value" does not work request: add --hydra to override hydra arguments (and if this is added the first one is not needed)Is that correct?
I used an env variable to avoid creating and endless loop of init/enqueue (using an argument like clearml.queue that would be captured and forwarded to the agent)
No luck with --args overrides="['dataset.path=abcd']"
--args param.ovveride=value
does not work with clearml-task
same result. in print(cfg) there are no overrides
Hmm can you try:--args overrides="['log.clearml=True','train.epochs=200','clearml.save=True']"
TrickyFox41 are you saying that if you add Task.init inthe code it works, but when you are calling "clearml-task" it does not work? (in both cases editing the Args/overrides ?
Hmm that should have worked ...
I'm assuming the Task itself is running on a remote agent, correct ?
Can you see the changes in the OmegaConf section ?
what happens when you pass--args overrides="['dataset.path=abcd']"
I tried like that clearml-task --script train.py --args overrides="log.clearml=True train.epochs=200 clearml.save=True" --project mornag-plain-dense --name mornag-plain-dense-training --queue tesla-t4 --skip-task-init
What do you have under Hydra section? and OmegaConf section
Yes the task is running on a remote agent with the --docker
flag
this is the config on the machine(s) running the agent
`
agent {
venvs_cache: {
max_entries: 50
free_space_threshold_gb: -1
path: ~/.clearml/venvs-cache
}
extra_docker_arguments: [
"--network", "host",
"-v", "/home/ubuntu/.ssh:/root/.ssh:ro",
"-v", "/home/ubuntu/.cache:/root/.cache",
]
docker_internal_mounts {
sdk_cache: "/clearml_agent_cache"
apt_cache: "/var/cache/apt/archives"
ssh_folder: "~/.ssh"
ssh_ro_folder: "/.ssh"
pip_cache: "/root/.pip-cache"
poetry_cache: "/root/.pypoetry-cache"
vcs_cache: "/root/.clearml/vcs-cache"
venv_build: "~/.clearml/venvs-builds"
pip_download: "/root/.clearml/pip-download-cache"
}
}
sdk {
aws {
s3 {
host: "...:443"
key: "..."
secret: "..."
secure: true
}
}
}
api {
web_server: https://...
api_server: https://...
file_server: https://...
credentials {
}
} `