Well, I assume ClearML SDK is waiting for models to be uploaded to the fileserver - is your self-hosted server remote, or on the same network? Can it be you have a limited bandwidth to it?

The fileserver is remote, but the bandwidth is not an issue.
Is the automatic artifact storage of clearml async ? (meaning even if the task is finished it could still be uploading associated artifacts ?)

yeah, it is async, but when talking a snapshot it will wait for the previous model to finish uploading I think

so if you have very large snapshots that are close to one another one might wait for the other for quite some time

Is there a way to make it synchronous ?

how are you reporting your models?

I don't really know. I just detected it automatically from the start, so I haven't looked into it yet.

Oh, if it's using the automagic than it's always in the background, which means it's also async

you can try calling task.flush(wait_for_uploads=True)

Okey thanks! I'll try this, if it does not work I'll just deactivate the automatic detection feature.