helper is returned as None for some reason

I do have write permissions

Is is even known if the bug is fixed on that version?

WebApp: 1.14.1-451 • Server: 1.14.1-451 • API: 2.28

maybe someone on your end can try to parse such a config and see if they also have the same problem

I need the zipping, chunking to manage millions of files

@<1523701070390366208:profile|CostlyOstrich36> It it still needed since Eugene thinks there is a bug?

Specifying it like this, gets me different error:

Exception has occurred: ValueError

• Insufficient permissions (delete failed) for None
  botocore.exceptions.ClientError: An error occurred (IllegalLocationConstraintException) when calling the DeleteObject operation: The me-south-1 location constraint is incompatible for the region specific endpoint this request was sent to.

During handling of the above exception, another exception occurred:

File "/home/ma...

well, I connected to mongodb manually and it is empty, loaded with just examples

I also see that elastisearch and mongo has some data

doesnt help

thanks for help

ok, is dataset path stored in mongo?
Im unable to find it in elasticsearch (debug images were here)

here is also another magic stuff

Bump, still waiting, closing in on a month since we are unable to deploy. We have team of 10+ people

I have tried:
Airflow - Pain to setup, old UI and other problems

Prefect - Literaly just tried to setup a simple distributed system, took me a week, I do not recommend this tool at all, horrible documentation, noone helps at slack.

Dagster - Absolute beauty, nice UI, easy to setup (as a pip package or just a docker + postgres), i highly recommend this tool. Takes a bit to get used to it. I will in coming week try this combo of dagster + clearml, where i periodically check some things and if...

We dont need a port
"s3" is part of url that is configured on our routers, without it we cannot connect

it looks like problem is the host field, whenever I add it I get:
2024-01-22 13:27:16,489 - clearml.storage - ERROR - Failed creating storage object None Reason: Missing key and secret for S3 storage access ( None )

we might as well have "s5" there but it is needed there

i need clearml.conf on my clearml server (in config folder which is mounted in docker-compose) or user PC? Or Both?
Its self hosted S3 thats all I know, i dont think it s Minio

ok, then, I have a solution, but it still makes duplicate names

• new_dataset._dataset_link_entries = {} # Cleaning all raw/a.png files
• resize a.png and save it in another location named a_resized.png
• Add back other files i need (excluding raw/a.png), I add them to new_dataset._ dataset_link_entries
• Use add_external_files to include it in dataset. Im also using dataset_path=[a list of relative paths]
  What I would expect:
  100 Files removed (all a.png)
  100 Files added (all a_resized.png)
  ...

When I look at LinkEntry object, link property is correct, no duplicates. Its relative_path thats duped and also key name in _dataset_link_entries

No, i specify where to upload
I see the data on S3 bucket is beeing uploaded. Just the log messages are really confusing

is there any way to see if I even have the data in mongodb?

• Here is how client side clearml.conf looks like together with the script im using to create the tasks. Uploads seems to work and is fixed thanks to you guys 🙌
  
  
  

We fixed the issue, thanks, had to update everything to latest.

Im pretty sure

But it seems like the data is gone, not sure how to get them back

I hope that its all the experiments