Sure:

def main():
    repo = "redacted"
    commit = "redacted"
    commit = "redacted"
    bands = ["redacted"]
    test_size = 0.2
    batch_size = 64
    num_workers = 12
    img_size = (128, 128)
    random_seed = 42
    epoch = 20
    learning_rate = 0.1

    livbatch_list = get_livbatch_list(repo, commit)
    lbs = download_batches(repo, commit, livbatch_list)

    df, label_map = get_annotation_df(lbs, bands)

    df_train, df_val = deterministic_train_val(df, test_size=test_siz...

Looks like it's because I did not do the mmpretrain way with dict config file ...

will do

@<1523701070390366208:profile|CostlyOstrich36> I would like to point to azure blob storage, what kind of url schema should I use ? And also, where do you configure the credential for the ClearML server to access to Azure blob as file_server ? I couldn't find any documentation around this topic 😞
TIA

you should be able to test your credential first using something like rclone or azure-cli

do you have a video showing the use case for clearml-session ? I struggle a bit about how is it used for ?

what is the difference between vscode via clearml-session and vscode via remote ssh extension ?

wow , did not know that vscode have a http "interface" !!! Make kind of sense as vscode is just a Chrome rendering webpage behind the scene ?

just saw that repo: who are coder ? That not the vscode developer team is it ?

Ok I think I found the issue. I had to point the file server to azure storage:

api {
    # Notice: 'host' is the api server (default port 8008), not the web server.
    api_server: 

    web_server: 

    files_server: "

"
    credentials {"access_key": "REDACTED", "secret_key": "REDACTED"}
}

@<1523701868901961728:profile|ReassuredTiger98> I found that you an set the file_server in your local clearml.conf to your own cloud storage. In our case, we use something like this in our clearml.conf:

api {
   file_server: "azure://<account>..../container"
}

All non artifact model are then store in our azure storage. In our self-hosted clearml setup, we don't even have a file server running alltogether

normally, you should have a agent running behind a "services" queue, as part of your docker-compose. You just need to make sure that you populate the appropriate configuration on the Server (aka set the right environment variable for the docker services)
That agent will run as long as your self-hosted server is running

but afaik this only works locally and not if you run your task on a clearml-agent!

Isn;t the agent using the same clearml.conf ?
We have our agent running task and uploading everything to Cloud. As I said, we don;t even have file server running

nope, we are self-hosted in Azure

no. I set apo.file_server to the None in Both the remote agent clearml.conf and my local clearml.conf
In which case, both case where the code is ran from local or remote, will store metrics to cloud storage

right, in which case you want to dynamically change with your code, not with the config file. This is where the Logger.set_default_output_upload come in

oh ..... did not know about that ...

you may want to share your config (with credential redacted) and the full docker compose start up log ?

Can you paste here what inside "Installed package" to double check ?

not sure how that work with Docker and machine that is not set up with ssh public key ... We will go to that path sometime in the future so I am quite interested too, on how people do it without ssh public key

with ssh public key, if from a terminal, I can do git clone, then so do the clearml agent, as it run on behalf of an local user. That apply to both local and VM

(I never played with pipeline feature so I am not really sure that it works as I imagined ...)

Do I need not make changes into clearml.conf so that it doesn't ask for my credentials or is there another way around

You have 2 options:

• set credential inside cleaml.conf : i am not familiar with this and never test it.
• or setup password less ssh with public key None

because when I was running both agents on my local machine everything was working perfectly fine

This is probably you (or someone) had set up ssh public key with your git repo sometime in the past

had you made sure that the agent inside GCP VM have access to your repository ? Can you ssh into that VM and try to do a git clone ?

one the same or different machine !

i need to do a git clone

You need to do it to test if it works. Clearml-agent will run it itself when it take in a task

yes

so the issue is that for some reason, the pip install by the agent don't behave the same way as your local pip install ?
Have you tried to manually install your module_b with pip install inside the machine that is running clearml-agent ? Seeing your example, looks like you are even running inside docker ?

@<1523701087100473344:profile|SuccessfulKoala55> Yes, I am aware of that one. It build docker container ... I wanted to build without docker. Like when clearml-agent run in non-docker mode, it is already building the running env inside it caching folder structure. I was wondering if there was a way to stop that process just before it execute the task .py