When I look at LinkEntry object, link property is correct, no duplicates. Its relative_path thats duped and also key name in _dataset_link_entries

We had a similar problem. Clearml doesnt support data migration (not that I know of)
So you have two ways to fix this:

• Recreate the dataset when its already in Azure
• Edit each elasticsearch database file entry to point to new destination (we did this)

ok, slight update. It seems like artifacts are uploading now to bucket. Maybe my folder explorer used old cache or something.
However, reported images are uploaded to fileserver instead of s3

here is the script im using to test things. Thanks

@<1523701070390366208:profile|CostlyOstrich36> It it still needed since Eugene thinks there is a bug?

we might as well have "s5" there but it is needed there

@<1523701070390366208:profile|CostlyOstrich36> Hello John, we are still unable to use clearml with our self hosted s3 CEPH instances, is there any update on the hotfix for 1.14?

I know these keys work, url and everything else works because I use these creds daily

No, i specify where to upload
I see the data on S3 bucket is beeing uploaded. Just the log messages are really confusing

removing it doesnt fix the problem

@<1523701070390366208:profile|CostlyOstrich36> Hello, im still unable to understand how to fix this

2. This is how web UI configurations looks like
   

clearml.conf is a fresh one i did clearml-init to make sure

@<1523701070390366208:profile|CostlyOstrich36> Still unable to understand what im doing wrong.
We have self hosted S3 Ceph storage server
Setting my config like this breaks task.init

@<1523701435869433856:profile|SmugDolphin23> Any news?

we use Ceph Storage Cluster, interface to it is the same as S3
I dont get what I have misconfigured.
The only thing I have not added is "region" field in clearml.conf because we literally dont have, its a self hosted cluster.
You can try and replicate this s3 config I have posted earlier.

unable to see the images with that link tho

Is is even known if the bug is fixed on that version?

WebApp: 1.14.1-451 • Server: 1.14.1-451 • API: 2.28

How can I do that?
I need to save the original hash, otherwise I lose all trackability to about 2k experiments

Yes, credetials seems to work
Im trying to figure out not why I dont see the uploaded files / folders

• I checked maybe clearml task uses fileserver instead but i dont see any files in fileserver folder
• Nothing is uploaded in bucket (i will ask IT guy to check if im uploading any files in logs)
  

@<1523701070390366208:profile|CostlyOstrich36> Any news on this? We are currently stuck without this fix, cant finish up clearml setup

The problem is that clearml.conf s3 config doesnt support empty region field, even empty strings crashes it

also, when uploading artifacts, I see where they are stored on the s3 bucket, but I cant find where the debug images are stored at

I tried it with port, but still having the same issue
Tried it with/without secure and multipart

We dont need a port
"s3" is part of url that is configured on our routers, without it we cannot connect

What you want is to have a service script that cleans up archived tasks, here is what we used: None

It looks like im moving forward

Setting url in clearml.conf without "s3" as suggested works (But I dont add port ther, not sure if it breaks something, we dont have a port)
host: " our-host.com "

Then in test_task.py
task: clearml.Task = clearml.Task.init(
project_name="project",
task_name="task",
output_uri=" None ",
)

I think connection is created
What im getting now is bucket error, i suppose I have to specify it so...

we do, yes. Changing it to https in settings doesnt help

ok, then, I have a solution, but it still makes duplicate names

• new_dataset._dataset_link_entries = {} # Cleaning all raw/a.png files
• resize a.png and save it in another location named a_resized.png
• Add back other files i need (excluding raw/a.png), I add them to new_dataset._ dataset_link_entries
• Use add_external_files to include it in dataset. Im also using dataset_path=[a list of relative paths]
  What I would expect:
  100 Files removed (all a.png)
  100 Files added (all a_resized.png)
  ...