Well, I assume ClearML SDK is waiting for models to be uploaded to the fileserver - is your self-hosted server remote, or on the same network? Can it be you have a limited bandwidth to it?

The fileserver is remote, but the bandwidth is not an issue.
Is the automatic artifact storage of clearml async ? (meaning even if the task is finished it could still be uploading associated artifacts ?)

yeah, it is async, but when talking a snapshot it will wait for the previous model to finish uploading I think

so if you have very large snapshots that are close to one another one might wait for the other for quite some time

Is there a way to make it synchronous ?

I don't really know. I just detected it automatically from the start, so I haven't looked into it yet.

how are you reporting your models?

Okey thanks! I'll try this, if it does not work I'll just deactivate the automatic detection feature.

you can try calling task.flush(wait_for_uploads=True)

Oh, if it's using the automagic than it's always in the background, which means it's also async