there is a typing in clearm.conf i sent you on like 87, there should be "key" not "ey" im aware of it

py file:
task: clearml.Task = clearml.Task.init(
project_name="project",
task_name="task",
output_uri=" None ",
)

clearml.conf:
{
# This will apply to all buckets in this host (unless key/value is specifically provided for a given bucket)
host: " our-host.com "
key: "xxx"
secret: "xxx"
multipart: false
...

• Here is how client side clearml.conf looks like together with the script im using to create the tasks. Uploads seems to work and is fixed thanks to you guys 🙌
  
  
  

WebApp: 1.16.0-494 • Server: 1.16.0-494 • API: 2.30
But be careful, upgrading is extremely dangerous

has 8 cores, so nothing fancy even

@<1523701601770934272:profile|GiganticMole91> Thats rookie numbers. We are at 228 GB for elastic now

hi, thanks for reaching out. Getting desperate here.
Yes, its self hosted
No, only currently running experiments are deleted (task itself is gone, but debug images and models are present in fileserver folder)

What I do see is some random elastisearch errors popping up from time to time

[2024-01-05 09:16:47,707] [9] [WARNING] [elasticsearch] POST None ` [status:N/A requ...

1. is 50GB elastisearch normal? Have you seen it. elsewhere or are we doing something wrong, one thing I think is that we are probably logging too frequently
2. Is it possible to somehow clean up this?

I also have noticed that this incident usually happens in the morning at around 6-7AM
Are there maybe some clearnup tasks or backups running on clearml server at those times?

Is that supposted to be so? How to fix it?

maybe someone on your end can try to parse such a config and see if they also have the same problem

will it be appended in clearml?
"s3" is part of domain to the host

We had a similar problem. Clearml doesnt support data migration (not that I know of)
So you have two ways to fix this:

• Recreate the dataset when its already in Azure
• Edit each elasticsearch database file entry to point to new destination (we did this)

The incident happened last friday (5 january)
Im giving you logs from around that time

I dont have a region. I guess I will wait till tomarrow then?

@<1523701435869433856:profile|SmugDolphin23> Setting it without http is not possible as it auto fills them back in

also, when uploading artifacts, I see where they are stored on the s3 bucket, but I cant find where the debug images are stored at

i need clearml.conf on my clearml server (in config folder which is mounted in docker-compose) or user PC? Or Both?
Its self hosted S3 thats all I know, i dont think it s Minio

Getting errors in elastisearch when deleting tasks, get retunred "cant delete experiment"

@<1523701070390366208:profile|CostlyOstrich36> Hello, im still unable to understand how to fix this

Is is even known if the bug is fixed on that version?

ClearML team should really write up some tutorial about this. I see this question weekly now. The short answer on what we did when we migrated servers was to wite a python script that takes data from clearml mongodb(stores tasks and datasets) and elastic (stores debug image urls, logs, scalars) and migrate them to other clearml instance databases

I purged all docker images and it still doesnt seem right
I see no side panel and it doesnt ask for login name

it looks like problem is the host field, whenever I add it I get:
2024-01-22 13:27:16,489 - clearml.storage - ERROR - Failed creating storage object None Reason: Missing key and secret for S3 storage access ( None )

Here are my clearml versions and elastisearch taking up 50GB

It is also possible to just make a copy of all the database files and move them to another server

Can I do it while i have multiple ongoing training?

i also think that if my package manager is set to uv, then it should only use uv and ignore pip at all