Artifact can be anything, that you can use clearml SDK to upload to storage. Which storage is used is defined by your clearml.conf (with its credentials) ClearML web and api server do not store those files

Model is a special artifact: None
Example you have the lineage feature where if you train model B using model A as starting point (aka pre-trained) , and model C from model B, ... The lineage will track modelC was built on...

is this mongodb type of filtering?

For #2: it's a pull rather than a push system: you need to have a script that do pulling at regular interval and need to keep track what new and what not?

something like this: None ?

@<1523701070390366208:profile|CostlyOstrich36>

Ok I think I found the issue. I had to point the file server to azure storage:

api {
    # Notice: 'host' is the api server (default port 8008), not the web server.
    api_server: 

    web_server: 

    files_server: "

"
    credentials {"access_key": "REDACTED", "secret_key": "REDACTED"}
}

I don;t think ClearML is designed to handle secrets other than git and storage ...

so it's not suppose to say "illegal output destination ..." ?

@<1523701070390366208:profile|CostlyOstrich36> Is there a way to tell clearml to not try to detect the Installed package ?

Oh, so you mean CLEARML_AGENT_SKIP_PIP_VENV_INSTALL=/path/to/my/venv/bin/python3 ??

So we have 3 python package, store in github.com
On the dev machine, the datascientist (DS) will add the local ssh key to his github account as authorized ssh keys, account level.
With the DS can run git clone git@github.com:org/repo1 then install that python package via pip install -e .
Do that for all 3 python packages, each in its own repo1 , repo2 and repo3 . All 3 can be clone using the same key that the DS added to his account.
The DS run a tra...

Should I put that in the clearml.conf file?

So I now just need to find a way to not populate Installed Package in the first place

Found a trick to have empty Installed package:
clearml.Task.force_requirements_env_freeze(force=True,requirements_file="/dev/null")
Not sure if this is the right way or not ...

So I tried:

CLEARML_AGENT_SKIP_PIP_VENV_INSTALL=/data/hieu/opt/python-venv/fastai/bin/python3.10
clearml-agent  daemon  --queue no_venv

Then enqueue a cloned task to no_venv
It is still trying to create a venv (and fail):

[...]
tag = 
docker_cmd = 
entry_point = debug.py
working_dir = apple_ic
created virtual environment CPython3.10.10.final.0-64 in 140ms
  creator CPython3Posix(dest=/data/hieu/deleteme/clearml-agent/venvs-builds/3.10, clear=False, no_vcs_ignore=False, gl...

But then how did the agent know where is the venv that it needs to use?

oh, looks like I need to empty the Installed Package before enqueue the cloned task

Done: None

not sure ... providing Zscaler certificate seems to allow clearml to talk to our clearml server, hosted in azure, Task init worked. But then failed to connect to the storage account (Azure too) ...

@<1523701087100473344:profile|SuccessfulKoala55> Thanks. Manage to get it working now with

export REQUESTS_CA_BUNDLE=/etc/ssl/certs/zscaler.crt

(Ubuntu system)

I will try it. But it's a bit random when this happen so ... We will see

you should be able to see int the Console tab that show what is happening

1.12.2 because some bug that make fastai lag 2x
1.8.1rc2 because it fix an annoying git clone bug

normally when new package need to be install, it shows up in the Console tab

clearml==1.12.2
clearml_agent v1.8.1rc2

@<1523701087100473344:profile|SuccessfulKoala55> I managed to make this working by:
concat the existing OS ca bundle and zscaler certificate. And set REQUESTS_CA_BUNDLE to that bundle file

Is it because Azure is "whitelisted" in our network ? Thus need a different certificate ?? And how do I provide 2 differents certificate ? Is bundling them simple as a concat of 2 pem file ?

from the logs, it feels like after git clone, it spend minutes without outputting anything. @<1523701205467926528:profile|AgitatedDove14> Do you know what is the agent suppose to do after git clone ?
I guess a check that all packages is installed ? But then with CLEARML_AGENT_SKIP_PYTHON_ENV_INSTALL=1, what is the agent doing ??

I know that git clone and pip verify all installed is normal. But for some reason in Michael screenshot, I don't see those steps ...

that agent is purely optional