I think you can set this code wise as well - https://clear.ml/docs/latest/docs/references/sdk/task#taskforce_requirements_env_freeze
Or you can just load a config file or object: https://clear.ml/docs/latest/docs/references/sdk/task/#connect_configuration
Or you can give it a configuration object: https://clear.ml/docs/latest/docs/clearml_sdk/task_sdk#configuration-objects
Maybe this opens up another question, which is more about how clearml-agent is supposed to be used. The "pure" way would be to make the docker image provide everything and clearml-agent should do not setup at all.
What I currently do instead is letting the docker image provide all system dependencies and let clearml-agent setup all the python dependencies. This allows me to reuse a docker image for more different experiments. However, then it would make sense to have as many configs as possible task-specific or task-overridable.
Hi ReassuredTiger98 !
I'm not sure the above will work. Maybe I can help in another way though: when you want to set
agent.package_manager.system_site_packages = true does that mean you have a docker container with some of the correct packages installed? In case you use a docker container, there is little no real need to create a virtualenv anyway and you might use the env var
CLEARML_AGENT_SKIP_PIP_VENV_INSTALL=1 to just install all packages in the root environment.
Because every task gets its own new clean docker container, there is no problem with using the root env. The nice thing is that in that way you get the system packages + any other ones that are installed by the Task.
Was that the outcome you meant? If so, please let me know when you tested it if it works for you 🙂 In the meantime we can think about your idea of making
ReassuredTiger98 anything in the configuration file can be overruled 🙂
no this should work with this one. I'll double check if I'm remembering it correctly but I thought you should be able to start a task after loading your own configuration object, where can set the
agent.package_manager.system_site_packages = true .
ReassuredTiger98 you can set different parameters per task: