Well, I assume ClearML SDK is waiting for models to be uploaded to the fileserver - is your self-hosted server remote, or on the same network? Can it be you have a limited bandwidth to it?
The fileserver is remote, but the bandwidth is not an issue.
Is the automatic artifact storage of clearml async ? (meaning even if the task is finished it could still be uploading associated artifacts ?)
yeah, it is async, but when talking a snapshot it will wait for the previous model to finish uploading I think
so if you have very large snapshots that are close to one another one might wait for the other for quite some time
I don't really know. I just detected it automatically from the start, so I haven't looked into it yet.
Okey thanks! I'll try this, if it does not work I'll just deactivate the automatic detection feature.
you can try calling task.flush(wait_for_uploads=True)
Oh, if it's using the automagic than it's always in the background, which means it's also async