I need the zipping, chunking to manage millions of files

I hope that its all the experiments

How can I do that?
I need to save the original hash, otherwise I lose all trackability to about 2k experiments

I already found the source code and i modified it as needed.
How can I now get this info from Task that is created when Dataset is created?
Couldnt find anything like clearml.Dataset(id=id).get_size()

i need clearml.conf on my clearml server (in config folder which is mounted in docker-compose) or user PC? Or Both?
Its self hosted S3 thats all I know, i dont think it s Minio

I also dont have side panel for some reason

Sounds similar to our issue? We have self hosted S3
None

No, i specify where to upload
I see the data on S3 bucket is beeing uploaded. Just the log messages are really confusing

I see the debug images in fileserver folder

1. is 50GB elastisearch normal? Have you seen it. elsewhere or are we doing something wrong, one thing I think is that we are probably logging too frequently
2. Is it possible to somehow clean up this?

digging deeper it seems like a parsing issue

This is the link generated

Is that supposted to be so? How to fix it?

there is a typing in clearm.conf i sent you on like 87, there should be "key" not "ey" im aware of it

I do notice another strange thing
Agent-services is down because It has no API key to clearm

clearml.conf is a fresh one i did clearml-init to make sure

elastisearch also takes like 15GB of ram

Elaseticsearch spazzing out

also, when uploading artifacts, I see where they are stored on the s3 bucket, but I cant find where the debug images are stored at

hi, thanks for reaching out. Getting desperate here.
Yes, its self hosted
No, only currently running experiments are deleted (task itself is gone, but debug images and models are present in fileserver folder)

What I do see is some random elastisearch errors popping up from time to time

[2024-01-05 09:16:47,707] [9] [WARNING] [elasticsearch] POST None ` [status:N/A requ...

@<1523701087100473344:profile|SuccessfulKoala55> Anything on this?

What do you mean by reusing the task for clearml Dataset, got a code example?
We have multiple different projects with multiple people working on each project.
This is our most used code on dataset uploading

ok, slight update. It seems like artifacts are uploading now to bucket. Maybe my folder explorer used old cache or something.
However, reported images are uploaded to fileserver instead of s3

here is the script im using to test things. Thanks

What you want is to have a service script that cleans up archived tasks, here is what we used: None

we do, yes. Changing it to https in settings doesnt help

we are cleaning, but there is a major problem
When deleting a task from web UI, nothing is deleted elsewhere
Debug images are not deleted, models are not deleted. And I suspect that scalars and logs are not deleted too
Im not sure why is that so

is there any way to see if I even have the data in mongodb?

py file:
task: clearml.Task = clearml.Task.init(
project_name="project",
task_name="task",
output_uri=" None ",
)

clearml.conf:
{
# This will apply to all buckets in this host (unless key/value is specifically provided for a given bucket)
host: " our-host.com "
key: "xxx"
secret: "xxx"
multipart: false
...