I do have write permissions

digging deeper it seems like a parsing issue

Here are my clearml versions and elastisearch taking up 50GB

ClearML team should really write up some tutorial about this. I see this question weekly now. The short answer on what we did when we migrated servers was to wite a python script that takes data from clearml mongodb(stores tasks and datasets) and elastic (stores debug image urls, logs, scalars) and migrate them to other clearml instance databases

It is also possible to just make a copy of all the database files and move them to another server

Bump, still waiting, closing in on a month since we are unable to deploy. We have team of 10+ people

Not really, but i think i will figure out the uv caching

I have another question @<1523701070390366208:profile|CostlyOstrich36>
How can i make the clearml agent to just run the image with just the uv
dont install any packages, nothing
i found docker_init_bash_script in clearml.config
i know there are some envs to pass in task init but that does not fully do what i want - just simply run the image, i have all the dependencies

@<1523701482157772800:profile|AnxiousSeal95> I see a lot of people here migrating data from one data source to another.
For us it was that we experimented with Clearml to get the feeling and we used clearml built in file storage to save debug images an all other artifacts.

Then we grew rapidly and we had to migrate to S3 storage.
I had to write a script that goes through elasticsearch and mongo db to point to new S3 links wher the data was migrated to.
I do however understand that migration...

Where can i override this so that it uses uv instead of trying to install python with apt

Can I do it while i have multiple ongoing training?

Yes, but does add_external_files makes chunked zips as add_files do?

Is it possible to split the large elasticsearch indexes? I know elasticsearch has something called rollover, but im not sure that clearml supports this

ok, is dataset path stored in mongo?
Im unable to find it in elasticsearch (debug images were here)

maybe someone on your end can try to parse such a config and see if they also have the same problem

You can check out boto3 python client (This is what we use to download / upload all S3 stuff), but minio-client probably already uses it under the hood.
We also use aws cli to do some downloading, it is way faster than python.

Regarding pdfs, yes, you have no choice but to preprocess it

WebApp: 1.16.0-494 • Server: 1.16.0-494 • API: 2.30
But be careful, upgrading is extremely dangerous

how to get rid of this auto appended line

i also think that if my package manager is set to uv, then it should only use uv and ignore pip at all

We had a similar problem. Clearml doesnt support data migration (not that I know of)
So you have two ways to fix this:

• Recreate the dataset when its already in Azure
• Edit each elasticsearch database file entry to point to new destination (we did this)

has 8 cores, so nothing fancy even

Is fileserver folder needed for successful backup?

thanks for help

is there any way to see if I even have the data in mongodb?

yes

This is what I see on fresh clearml
Where all my mounts are on /mnt/data/clearml-server instead of /opt/clearml

I get sidebars and login on my local PC
But the data isnt loaded

I tried to not edit anything in docker-compose and just paste my data in there. Didnt help

here is also another magic stuff

well, I connected to mongodb manually and it is empty, loaded with just examples