@<1523701070390366208:profile|CostlyOstrich36> It it still needed since Eugene thinks there is a bug?

ok, is dataset path stored in mongo?
Im unable to find it in elasticsearch (debug images were here)

im also batch uploading, maybe thats the problem?

• The dataset is about 1TB containing 1 million files
• I dont have the SSD space locally to do the upload
• So i download a part of the dataset, use add_files() and then upload() to that batch
• Upload the dataset
  I noticed that each batch is slower and slower

Our datasets are more than 1TB in size and will grow in size (probably 4TB and up), this means we also need 4TB local storage just to upload the dataset back in zipped format. This is not a good solution.

What we can do I guess is do the downloading locally by some chunks of files?
Download locally 100 files, add_to_clearml dataset, repeat

hi, thanks for reaching out. Getting desperate here.
Yes, its self hosted
No, only currently running experiments are deleted (task itself is gone, but debug images and models are present in fileserver folder)

What I do see is some random elastisearch errors popping up from time to time

[2024-01-05 09:16:47,707] [9] [WARNING] [elasticsearch] POST None ` [status:N/A requ...

We fixed the issue, thanks, had to update everything to latest.

we might as well have "s5" there but it is needed there

digging deeper it seems like a parsing issue

i need clearml.conf on my clearml server (in config folder which is mounted in docker-compose) or user PC? Or Both?
Its self hosted S3 thats all I know, i dont think it s Minio

will it be appended in clearml?
"s3" is part of domain to the host

@<1523701070390366208:profile|CostlyOstrich36> Hello, im still unable to understand how to fix this

Bump, still waiting, closing in on a month since we are unable to deploy. We have team of 10+ people

@<1523701435869433856:profile|SmugDolphin23> Any news?

also, if I try to set the url to None it auto replaces it with None : None

We dont need a port
"s3" is part of url that is configured on our routers, without it we cannot connect

Is that supposted to be so? How to fix it?

It looks like im moving forward

Setting url in clearml.conf without "s3" as suggested works (But I dont add port ther, not sure if it breaks something, we dont have a port)
host: " our-host.com "

Then in test_task.py
task: clearml.Task = clearml.Task.init(
project_name="project",
task_name="task",
output_uri=" None ",
)

I think connection is created
What im getting now is bucket error, i suppose I have to specify it so...

So from our IT guys i now know that
"s3" part of url is subdomain, we use it in all other libs like boto3 and cloudpathlib, never had any problems
This is where the crash happens inside the clearml Task

1. is 50GB elastisearch normal? Have you seen it. elsewhere or are we doing something wrong, one thing I think is that we are probably logging too frequently
2. Is it possible to somehow clean up this?

elastisearch also takes like 15GB of ram

I already found the source code and i modified it as needed.
How can I now get this info from Task that is created when Dataset is created?
Couldnt find anything like clearml.Dataset(id=id).get_size()

@<1523701435869433856:profile|SmugDolphin23> Any ideas how to fix this?

No, i specify where to upload
I see the data on S3 bucket is beeing uploaded. Just the log messages are really confusing

we are cleaning, but there is a major problem
When deleting a task from web UI, nothing is deleted elsewhere
Debug images are not deleted, models are not deleted. And I suspect that scalars and logs are not deleted too
Im not sure why is that so

there is a typing in clearm.conf i sent you on like 87, there should be "key" not "ey" im aware of it

after task is deleted

also, when uploading artifacts, I see where they are stored on the s3 bucket, but I cant find where the debug images are stored at