not sure how that work with Docker and machine that is not set up with ssh public key ... We will go to that path sometime in the future so I am quite interested too, on how people do it without ssh public key

with ssh public key, if from a terminal, I can do git clone, then so do the clearml agent, as it run on behalf of an local user. That apply to both local and VM

I use ssh public key to access to our repo ... Never tried to provide credential to clearml itself (via clearml.conf ) so I cannot help much here ...

you should be able to use as many agent as you want.
On the same or different queue

one the same or different machine !

there is a whole discussion about it here: None

(I never played with pipeline feature so I am not really sure that it works as I imagined ...)

had you made sure that the agent inside GCP VM have access to your repository ? Can you ssh into that VM and try to do a git clone ?

Nevermind: None

By default, the File Server is not secured even if Web Login Authentication has been configured. Using an object storage solution that has built-in security is recommended.

My bad

For #2: it's a pull rather than a push system: you need to have a script that do pulling at regular interval and need to keep track what new and what not?

what is the difference between vscode via clearml-session and vscode via remote ssh extension ?

wow , did not know that vscode have a http "interface" !!! Make kind of sense as vscode is just a Chrome rendering webpage behind the scene ?

Are you sure all the files needed are pushed to your git repo ?
Go to a another folder and git clone that exact branch/commit and check the files are there ?

please provide the full logs and error message.

what about the log aroundwhen it try to actually clone your repo ?

very hard to diagnose with this tiny bit of log ...

or simply create a new venv in your local PC, then install your package with pip install from repo url and see if your file is deployed properly in that venv

once you install manually your package inside the docker container, check that your file module_b/templates/my_template.yml is where it should be

Just keep in mind my your bottleneck will be the transfer rate. So mounting will not save you anything as you still need to transfer the whole dataset sooner or later to your GPU instance.
One solution is as Jake suggest. The other can be pre-download the data to your instance with a CPU only cheap instance type, then restart the instance with GPU.

you should be able to test your credential first using something like rclone or azure-cli

Try to set CLEARML_AGENT_SKIP_PYTHON_ENV_INSTALL=true in the terminal start clearml-agent
See None

Hi.
How do you tell the server to use my azure storage instead of local drive, on the host machine ? Isn't it by setting azure.storage in /opt/clearml/config/clearml.conf ?

Ok I think I found the issue. I had to point the file server to azure storage:

api {
    # Notice: 'host' is the api server (default port 8008), not the web server.
    api_server: 

    web_server: 

    files_server: "

"
    credentials {"access_key": "REDACTED", "secret_key": "REDACTED"}
}

We don't have a file server. The clearml conf have :
sdk.development.default_output_uri=" None "

In the web UI, in the queue/worker tab, you should see a service queue and a worker available in that queue. Otherwise the service agent is not running. Refer to John c above

but then it still missing a bunch of library in the Taks (that succeed) > Execution > INSTALLED PACKAGES
So when I do a clone of that task, and try to run the clone, the task fail because it is missing python package 😞

(wrong tab sorry :P)

so it's not suppose to say "illegal output destination ..." ?

that format is correct as I can run pip install -r requirements.txt
using the exact same file

is task.add_requirements("requirements.txt") redundant ?
Is ClearML always look for a requirements.txt in the repo root ?