It's also sufficient to see StorageManager.list("/data/clear") takes a really long time to return no results

True, and we plan to migrate to pipelines once we have some time for it :) but anyway that condition is flawed I believe

Ubuntu 18.04, latest clearml version

https://github.com/allegroai/clearml/pull/533 🙂

The logs are on the bucket, yes.
The default file server is also set to s3://ip:9000/clearml

proj_suffix = "" i = 2 while Task.get_project_id(f"{proj_name}{proj_suffix}") is not None: tasks = Task.get_tasks(project_name=f"{proj_name}{proj_suffix}") if not [task for task in tasks if not task.get_archived()]: # Empty project, we can use this one... break proj_suffix = f"_{i}" i += 1

Yes

1.8.3; what about when calling task.close() ? We suddenly have a need to setup our logging after every task.close() call

See e None @<1523701087100473344:profile|SuccessfulKoala55>

Can I subscribe to updates? 😁

The api.files_server is set to the MinIO endpoint s3://ip:9000/clearml (both locally and remotely) The sdk.development.default_output_uri is set to the MinIO endpoint (both locally and remotely) When we call Task.init I do not set the output_uri at all I get the logger directly with task.get_logger()

Don't even need to specify json=... 😉 Thanks!

clearml.backend_api.session.defs.ENV_HOST.get() did not work unfortunately 🤔

On it! Should I include the additional user filters described above?

Or the above;)

Added the following line under volumes for apiserver , fileserver , agent-services :
- /data/clearml:/data/clearml

I narrowed the bug down to the "fix" in 1.8.1, see my other post

After setting the sdk.development.default_output_uri in the configs, my code kinda looks like:
` task = Task.init(project_name=..., task_name=..., tags=...)
logger = task.get_logger()

report with logger freely `

Bump?

This also appears in the error log:

` StorageManager.download_folder(cache_dir.as_posix(), local_folder=".")
File "/home/idan/.clearml/venvs-builds/3.7/lib/python3.7/site-packages/clearml/storage/manager.py", line 278, in download_folder
for path in helper.list(prefix=remote_url):
File "/home/idan/.clearml/venvs-builds/3.7/lib/python3.7/site-packages/clearml/storage/helper.py", line 596, in list
res = self._driver.list_container_objects(self._container, ex_prefix=prefix)
Fi...

But it is strictly that if condition in Task.init, see the issue I opened about it

I couldn't find it directly in the SDK at least (in the APIClient)... 🤔

Let me try that real quick

Actually, it appears some elements (scalars, plots, etc) have no migrated by moving mongodb data.
Where are these stored? Any idea @<1523701827080556544:profile|JuicyFox94> ?

Generally the StorageManager seems a bit slow, even a simple StorageManager.list(...) on a local path seems to take a long time

Yeah that works fine 😄 I just fetch it once to map argparse users to their IDs for later filtering.

The Slack Monitoring example should be updated btw, as they now use slack_sdk instead of slack (in the import statements)

CostlyOstrich36 I'm not sure what you mean by "through the apps", but any script AFAICS would expose the values of these environment variables; or what am I missing?

~20

I'm not sure why internally ClearML tries to initialize a task when get_task is called...

Btw TimelyPenguin76 this should also be a good starting point:
First create the target directory and add some files:sudo mkdir /data/clearml sudo chmod 777 -R /data/clearml touch /data/clearml/foo touch /data/clearml/bar touch /data/clearml/bazThen list the files using the StorageManager. It shouldn't take more than a few miliseconds.` from clearml import StorageManager

%%timeit
StorageManager.list("/data/clearml")

-> 21.2 s ± 328 ms per loop (mean ± std. dev. of 7 runs, 1 loop each) `