@<1709740168430227456:profile|HomelyBluewhale47> We have the same problem. Millions of files, stored on CEPH. I would not recommend you to do it this way. Everything gets insanely slow (dataset.list_files, downloading the dataset, removing files)

The way I use Clearml Datasets for large number of samples now is to save a json which stores all paths to samples in Dataset metadata:
clearml_dataset.set_metadata(metadata, metadata_name=metadata_key)

However this then means that you need wrappe...

well, I connected to mongodb manually and it is empty, loaded with just examples

But it seems like the data is gone, not sure how to get them back

I guess I fucked up something when moving files

I get the same when I copy /opt/clearml/data folder into /mnt/data/clearml/data

I get sidebars and login on my local PC
But the data isnt loaded

I tried to not edit anything in docker-compose and just paste my data in there. Didnt help

Is fileserver folder needed for successful backup?

thanks for help

Log dump

Im basically trying to force the agent to use uv defined python

yes

Where can i override this so that it uses uv instead of trying to install python with apt

here is also another magic stuff

I see in clearml-agent that it is created here

doesnt help

No, i specify where to upload
I see the data on S3 bucket is beeing uploaded. Just the log messages are really confusing

Not really, but i think i will figure out the uv caching

I have another question @<1523701070390366208:profile|CostlyOstrich36>
How can i make the clearml agent to just run the image with just the uv
dont install any packages, nothing
i found docker_init_bash_script in clearml.config
i know there are some envs to pass in task init but that does not fully do what i want - just simply run the image, i have all the dependencies

I also dont have side panel for some reason

This is what I see on fresh clearml
Where all my mounts are on /mnt/data/clearml-server instead of /opt/clearml

clearml.conf is a fresh one i did clearml-init to make sure

I tried it with port, but still having the same issue
Tried it with/without secure and multipart

digging deeper it seems like a parsing issue

The problem is that clearml.conf s3 config doesnt support empty region field, even empty strings crashes it

we might as well have "s5" there but it is needed there

Bump, still waiting, closing in on a month since we are unable to deploy. We have team of 10+ people

I know these keys work, url and everything else works because I use these creds daily

What do you mean by reusing the task for clearml Dataset, got a code example?
We have multiple different projects with multiple people working on each project.
This is our most used code on dataset uploading

This is the link generated

we do, yes. Changing it to https in settings doesnt help

@<1523703436166565888:profile|DeterminedCrab71> Thanks for responding
It was unclear to me that I need to set 443 also everywhere in clearml.conf
Setting s3 host urls with 443 in clearml.conf and also in web UI made it work

Im now almost at the finish line. The last thing that would be great is to fix archived task deletion.
For some reason i have error of missing S3 keys in clearml docker compose logs, the folder / files are not deleted in S3 bucket.

You can see how storage_credentials.co...