while the other may need to be 1 instead of true

Solved @<1533620191232004096:profile|NuttyLobster9> . In my case:
I need to from clearml import Task very early in the code (like first line), before importing argparse
And not calling task.connect(parser)

you should know where your latest model is located then just call task.upload_artifact on that file ?

you should be able to explicitly upload a file of your choice as artefact using something like this: None

we are not using docker compose. We are deploying in Azure with each database as a standalone service

To "attach" that zip to the model, do you just use the update_weight and point to that zip file?

But then how did the agent know where is the venv that it needs to use?

Set that env var in the terminal before running the agent ?

can you make train1.py use clearml.conf.server1 and train2.py use clearml.conf2 ?? In which case I would be intersted @<1523701087100473344:profile|SuccessfulKoala55>

Oh, so you mean CLEARML_AGENT_SKIP_PIP_VENV_INSTALL=/path/to/my/venv/bin/python3 ??

oh, looks like I need to empty the Installed Package before enqueue the cloned task

None

Onprem: User management is not "live" as you need to reboot and password are hardcoded ... No permission distinction, as everyone is admin ...

following your example, if the seeds are hard coded in the code, then git hash will detect if changed happen and the step need to be run or not

so the issue is that for some reason, the pip install by the agent don't behave the same way as your local pip install ?
Have you tried to manually install your module_b with pip install inside the machine that is running clearml-agent ? Seeing your example, looks like you are even running inside docker ?

We don't have a file server. The clearml conf have :
sdk.development.default_output_uri=" None "

we are usign mmsegmentation by the way

Just keep in mind my your bottleneck will be the transfer rate. So mounting will not save you anything as you still need to transfer the whole dataset sooner or later to your GPU instance.
One solution is as Jake suggest. The other can be pre-download the data to your instance with a CPU only cheap instance type, then restart the instance with GPU.

right, in which case you want to dynamically change with your code, not with the config file. This is where the Logger.set_default_output_upload come in

you should be able to test your credential first using something like rclone or azure-cli

normally when new package need to be install, it shows up in the Console tab

So "Using env ..." take minutes without any output ?

I think a proper screenshot of the full log with some information redacted is the way to go. Otherwise we are just guessing in the dark

I don't use submodule so don't really know how that behave with ClearML

I can only guess with little information here. You better try to debug with print statement. Is this happening in submodule uncommited changes ?

Looks like your issue is not that ClearML is not tracking your changes but more about your Configuration is overwrriten.
This often happen to me. The way I debug this is put a lot of print statement along the code to track when the Configuration is overwriten and narrow down why. print statement will show up in the Console tab.

I think he means disk space usage ...

this look like you had configured git to track a bunch of file that should not be tracked like uv.lock or .DS_Store ...

We need to focus first on Why is it taking minutes to reach Using env.
In our case, we have a container that have all packages installed straight in the system, no venv in the container. Thus we don't use CLEARML_AGENT_SKIP_PIP_VENV_INSTALL
But then when a task is pulled, I can see all the steps like git clone, a bunch of Requirement already satisfied .... There may be some odd package that need to be installed because one of our DS is experimenting ... But all that we can see what is...

Hi.
How do you tell the server to use my azure storage instead of local drive, on the host machine ? Isn't it by setting azure.storage in /opt/clearml/config/clearml.conf ?

you should be able to see int the Console tab that show what is happening