so the issue is that for some reason, the pip install by the agent don't behave the same way as your local pip install ?
Have you tried to manually install your module_b with pip install inside the machine that is running clearml-agent ? Seeing your example, looks like you are even running inside docker ?

We don't have a file server. The clearml conf have :
sdk.development.default_output_uri=" None "

we are usign mmsegmentation by the way

Just keep in mind my your bottleneck will be the transfer rate. So mounting will not save you anything as you still need to transfer the whole dataset sooner or later to your GPU instance.
One solution is as Jake suggest. The other can be pre-download the data to your instance with a CPU only cheap instance type, then restart the instance with GPU.

right, in which case you want to dynamically change with your code, not with the config file. This is where the Logger.set_default_output_upload come in

you should be able to test your credential first using something like rclone or azure-cli

I think a proper screenshot of the full log with some information redacted is the way to go. Otherwise we are just guessing in the dark

I don't use submodule so don't really know how that behave with ClearML

I can only guess with little information here. You better try to debug with print statement. Is this happening in submodule uncommited changes ?

Looks like your issue is not that ClearML is not tracking your changes but more about your Configuration is overwrriten.
This often happen to me. The way I debug this is put a lot of print statement along the code to track when the Configuration is overwriten and narrow down why. print statement will show up in the Console tab.

this look like you had configured git to track a bunch of file that should not be tracked like uv.lock or .DS_Store ...

Hi.
How do you tell the server to use my azure storage instead of local drive, on the host machine ? Isn't it by setting azure.storage in /opt/clearml/config/clearml.conf ?

python library don't always use OS certificates ... typically, we have to set REQUESTS_CA_BUNDLE=/path/to/custom_ca_bundle_crt because requests ignore OS certificates

I really like how you make all this decoupled !! 🎉

I tried mounting azure storage account on that path and it worked: all files end up in the cloud storage

you may want to share your config (with credential redacted) and the full docker compose start up log ?

@<1523701070390366208:profile|CostlyOstrich36> I would like to point to azure blob storage, what kind of url schema should I use ? And also, where do you configure the credential for the ClearML server to access to Azure blob as file_server ? I couldn't find any documentation around this topic 😞
TIA

Nice ! That is handy !!
thanks !

you mean this dump section ?

Please refer to here None
The doc need to be a bit clearer: one require a path and not just true/false

If you care about the local destination then you may want to use this None

your need both in certain case

@<1523701087100473344:profile|SuccessfulKoala55> I can confirm that v1.8.1rc2 fixed the issue in our case. I manage to reproduce it:

• Do a local commit without pushing
• Create task and queue it
• The queue task failed as expected as the commit is only local
• Push your local commit
• Requeue the task
• Expecting that the task succeeed as the commit is avail: but it fails as the vcs seems to be in weird state from previous failure
• Now with v1.8.1rc2 the issue is solved

I saw that page ... but nothing about number of worker of a queue .... or did I miss it ?

Yes. I am investigating that route now.

I don;t think there is a "kill task" code. By principle, in Linux, as a parent process, ClearML agent launch the training process. When a parent process is terminated, the linux kernel will, in most of the case, kill all child processes, including your training process.
There may be some way to resume a task from ClearML agent when it restart, but I don;t think that is the default behavior

If the agent is the one running the experiment, very likely that your task will be killed.
And when the agent come back, immediately or later, probably nothing will happen. It won't resume ...

So I now just need to find a way to not populate Installed Package in the first place

very hard to diagnose with this tiny bit of log ...

hard to see with your croppout here an there ...