Gotcha, and the agent default runtime mode is docker correct? So I could install all my system dependencies in my own docker image?
I guess I'm confused on venv mode vs docker mode. It seems like I'm passing in my own docker image which is then used at run time?
basically the idea is you do not need to configure the Experiment manually, it is created when you actually develop the code / run/debug it, or you have the CLI taking everything from your machine and populating it
BoredHedgehog47 this is basically a wizard explaining the steps, see the 3 tabs 🙂
BTW, you can launch an experiment directly from CLI with clearml-task
https://clear.ml/docs/latest/docs/apps/clearml_task
seems like I'm passing in my own docker image which is then used at run time?
You are passing the Default docker image, if the Task does not list a specific docker image it will use the one you passed.
Yes this is "docker mode" (in venv mode no dockers are used, it just creates a new venv per experiment and installs everything inside the venv)
Yep got it, I was under the impression I could set those values in the UI but I now see they are parsed from my local workstation
btw: you can also configure --extra-index-url in the agent's clearml.conf
How does a task specify which docker image it needs?
Either in the code itself 'task.set_base_docker' or with the CLI, or set it in the UI when you clone an experiment (everything becomes editable)
Does the clearml module parse the python packages?
Yes it analyzes the installed packages based on the actual mports you have in the code.
If I'm using a private pypi artifact server, would I set the PIP_INDEX_URL on the workers so they could retrieve those packages when that experiment is cloned and re-ran?
Correct 🙂 the agent basically calls pip install on those packages, so if you configure it, with PIP_INDEX_URL it should just work like any other pip install
How does a task specify which docker image it needs?
and the agent default runtime mode is docker correct?
Actually the default is venv mode, to run in docker mode add --docker to the command line
So I could install all my system dependencies in my own docker image?
Correct, inside the docker it will inherit all the preinstalled packages, But it will also install any missing ones (based on the Task requirements. i.e. "installed packages" section)
Also what is the purpose of the
aws
block in the clearml.conf? Where are those values used?
Access to S3 buckets / credentials (passed to the internal StorageManager artifacts uploads etc)
Also what is the purpose of the aws block in the clearml.conf? Where are those values used?
Does the clearml module parse the python packages? If I'm using a private pypi artifact server, would I set the PIP_INDEX_URL on the workers so they could retrieve those packages when that experiment is cloned and re-ran?