So we have 3 python package, store in github.com
On the dev machine, the datascientist (DS) will add the local ssh key to his github account as authorized ssh keys, account level.
With the DS can run git clone git@github.com:org/repo1 then install that python package via pip install -e .
Do that for all 3 python packages, each in its own repo1 , repo2 and repo3 . All 3 can be clone using the same key that the DS added to his account.
The DS run a tra...

following this thread as it happen every now and then that clearml miss some package for some reason ...

my code looks like this :

    parser = argparse.ArgumentParser()
    parser.add_argument('-c', '--config-file', type=str, default='train_config.yaml',
                        help='train config file')
    parser.add_argument('-t', '--train-times', type=int, default=1,
                        help='train the same model several times')
    parser.add_argument('--dataset_dir', help='path to folder containing the preped dataset.', required=True)
    parser.add_argument('--backup', action='s...

I mean, depend on what do you want to report ... if you want to stick to table, I suggest earlier to gather your stats in table format ...
Otherwise, matplotlib seems to be the most user friendly way

I also use this: None
Which can give more control

you can upload the df as artifact.
Or the statistics as a DataFrame and upload as artifact ?

if you want plot, you can simply generate plot with matplotlib and clearml can upload them in the Plot or Debug Sample section

None

with

    df = pd.DataFrame({'num_legs': [2, 4, 8, 0],
                       'num_wings': [2, 0, 0, 0],
                       'num_specimen_seen': [10, 2, 1, 8]},
                      index=['falcon', 'dog', 'spider', 'fish'])

    import clearml
    task = clearml.Task.current_task()
    task.get_logger().report_table(title='table example', series='pandas DataFrame', iteration=0, table_plot=df)
    # logger.report_table(title='table example',series='pandas DataFrame',iteration=0,tabl...

and just came across this: None
That sounds like what you may be looking for

do you mean having the ClearML FileServer store on azure blob instead of on the local drive?

Yes, that is what I wanted.

If so, that's not possible. You can however point the fileserver data folder to some mounted folder - if you have something that can create a mount from a filesystem folder to azure blob, it will work (the file server will always treat it as a local file system)

Thanks for confirming that it's the only solution. 👍

Solved @<1533620191232004096:profile|NuttyLobster9> . In my case:
I need to from clearml import Task very early in the code (like first line), before importing argparse
And not calling task.connect(parser)

I will try it. But it's a bit random when this happen so ... We will see

@<1523701087100473344:profile|SuccessfulKoala55> I can confirm that v1.8.1rc2 fixed the issue in our case. I manage to reproduce it:

• Do a local commit without pushing
• Create task and queue it
• The queue task failed as expected as the commit is only local
• Push your local commit
• Requeue the task
• Expecting that the task succeeed as the commit is avail: but it fails as the vcs seems to be in weird state from previous failure
• Now with v1.8.1rc2 the issue is solved

I use ssh public key to access to our repo ... Never tried to provide credential to clearml itself (via clearml.conf ) so I cannot help much here ...

@<1523701087100473344:profile|SuccessfulKoala55> It's working !! Thank you very much !!! Clearml is awesome !!!!

Try to set CLEARML_AGENT_SKIP_PYTHON_ENV_INSTALL=true in the terminal start clearml-agent
See None

from what I understand, the docker mode were designed for apt based image and also running as root inside the container.
We have container that are not apt based and running not as root
We also do some "start up" that fetch credentials from Key Vault prior running the agent

you mean this dump section ?

had you made sure that the agent inside GCP VM have access to your repository ? Can you ssh into that VM and try to do a git clone ?

python library don't always use OS certificates ... typically, we have to set REQUESTS_CA_BUNDLE=/path/to/custom_ca_bundle_crt because requests ignore OS certificates

is this mongodb type of filtering?

Can you paste here what inside "Installed package" to double check ?

if you are on github.com , you can use Fine tune PAT token to limit access to minimum. Although the token will be tight to an account, it's quite easy to change to another one from another account.

how did you deploy your clearml server ?

(I never played with pipeline feature so I am not really sure that it works as I imagined ...)

Should I get all the workers None
Then go through them and count how many is in my queue of interest ?

got it

Thanks @<1523701070390366208:profile|CostlyOstrich36>

I saw that page ... but nothing about number of worker of a queue .... or did I miss it ?