Anyhow if the StorageManager.upload was fast, the upload_artifact is calling that exact function. So I don't think we actually have an issue here. What do you think?

Does StorageManager.upload and upload_artifact use the same methods?

Is the only difference is task being async?

Does StorageManager.upload and upload_artifact use the same methods?

Yes they both use StorageManager.upload

Is the only difference is task being async?

Two differences:
Upload being async Registering the artifact on the experiment. StorageManager will only upload, where as upload_artifact will make sure the file is registered as an artifact on the experiment, together with all of the artifacts properties.

Hmm

What if I register the artifact manually?

What if I register the artifact manually?

task.upload_artifact('local folder', artifact_object=' ')This one should be quite quick, it's updating the experiment

artifact_object is local file in my case. I did not understand what you suggest

upload_artifact will actually do two things:
upload the file to the trains-server register it as an artifact on the experiment
What did you mean by "register the artifact manually"? You still need to upload the file to the trains-server (so it is later accessible )

I was thinking to do 2. step manually

So you are saying that step 1 is the problematic one?

None of them is problematic, this is what I'm trying to say 🙂
I think the minio browser gets confused.
if you want to test the upload time on the client you can try:
task.flush(wait_for_uploads=True) tic = time() task.upload_artifact('test', '/tmp/localfile') task.flush(wait_for_uploads=True) print(time() - tic)

There is some overhead, but it should be negligible.

The code took 0.16 for 33 MB

That seems reasonable to me:)

Yep

okay we are good 🙂

So I need to fix Minio then?

No, I think it might be a glitch in the way the calculate the upload speed, nothing we can do 🙂

Whose glitch on calculating the upload speed then? What will I do to fix my problem? 😞

What will I do to fix my problem?

What is the problem? we just proved the upload speed is just fine?

The latest code you've posted uplods to local.
I need to upload to my local Minio

When I give my Minio to output_uri argument, it uploads 500 KB /sec as before.

When I give my Minio to output_uri argument, it uploads 500 KB /sec as before.

But it worked well when using StorageManager and uploading to the minio directly, is that correct?

.. I give my Minio to output_uri argument

How long did it take to run the demo code I posted?
(The one you mentioned took 0.16s to run locally)

MuddyCrab47 could you post the full sample code you are using?

` from trains import Task

task = Task.init(project_name="SpeedTest",
task_name="test1",
task_type=Task.TaskTypes.data_processing,
output_uri="s3://<my_minio_ip>:<my_minio_port>/<my_bucket_name>",
reuse_last_task_id=False)

task.upload_artifact(name="test1",
artifact_object="<my_local_file>") `

I think I found something, let me dig deeper 🙂

Thanks MuddyCrab47 !!!
I found it!
It turns out the artifact upload will always upload from stream (aka no multi-upload). I will make sure we fix it in the next RC (I think the plan is to have it out this weekend)

Oh, glad we found out a solution.

How about until weekend? Do you have a temporary solution=

MuddyCrab47 Try this one 🙂

Yes!
This solved the problem.
Thank you AgitatedDove14 !