https://github.com/allegroai/clearml/pull/533 🙂

The logs are on the bucket, yes.
The default file server is also set to s3://ip:9000/clearml

proj_suffix = "" i = 2 while Task.get_project_id(f"{proj_name}{proj_suffix}") is not None: tasks = Task.get_tasks(project_name=f"{proj_name}{proj_suffix}") if not [task for task in tasks if not task.get_archived()]: # Empty project, we can use this one... break proj_suffix = f"_{i}" i += 1

Yes

1.8.3; what about when calling task.close() ? We suddenly have a need to setup our logging after every task.close() call

See e None @<1523701087100473344:profile|SuccessfulKoala55>

Can I subscribe to updates? 😁

The api.files_server is set to the MinIO endpoint s3://ip:9000/clearml (both locally and remotely) The sdk.development.default_output_uri is set to the MinIO endpoint (both locally and remotely) When we call Task.init I do not set the output_uri at all I get the logger directly with task.get_logger()

Don't even need to specify json=... 😉 Thanks!

clearml.backend_api.session.defs.ENV_HOST.get() did not work unfortunately 🤔

On it! Should I include the additional user filters described above?

Or the above;)

Added the following line under volumes for apiserver , fileserver , agent-services :
- /data/clearml:/data/clearml

I narrowed the bug down to the "fix" in 1.8.1, see my other post

After setting the sdk.development.default_output_uri in the configs, my code kinda looks like:
` task = Task.init(project_name=..., task_name=..., tags=...)
logger = task.get_logger()

report with logger freely `

This also appears in the error log:

` StorageManager.download_folder(cache_dir.as_posix(), local_folder=".")
File "/home/idan/.clearml/venvs-builds/3.7/lib/python3.7/site-packages/clearml/storage/manager.py", line 278, in download_folder
for path in helper.list(prefix=remote_url):
File "/home/idan/.clearml/venvs-builds/3.7/lib/python3.7/site-packages/clearml/storage/helper.py", line 596, in list
res = self._driver.list_container_objects(self._container, ex_prefix=prefix)
Fi...

But it is strictly that if condition in Task.init, see the issue I opened about it

I couldn't find it directly in the SDK at least (in the APIClient)... 🤔

Let me try that real quick

Actually, it appears some elements (scalars, plots, etc) have no migrated by moving mongodb data.
Where are these stored? Any idea @<1523701827080556544:profile|JuicyFox94> ?

Yeah that works fine 😄 I just fetch it once to map argparse users to their IDs for later filtering.

The Slack Monitoring example should be updated btw, as they now use slack_sdk instead of slack (in the import statements)

CostlyOstrich36 I'm not sure what you mean by "through the apps", but any script AFAICS would expose the values of these environment variables; or what am I missing?

~20

Btw TimelyPenguin76 this should also be a good starting point:
First create the target directory and add some files:sudo mkdir /data/clearml sudo chmod 777 -R /data/clearml touch /data/clearml/foo touch /data/clearml/bar touch /data/clearml/bazThen list the files using the StorageManager. It shouldn't take more than a few miliseconds.` from clearml import StorageManager

%%timeit
StorageManager.list("/data/clearml")

-> 21.2 s ± 328 ms per loop (mean ± std. dev. of 7 runs, 1 loop each) `

Kinda, yes, and this has changed with 1.8.1.
The thing is that afaik currently ClearML does not officially support a remotely executed task to spawn more tasks, so we also have a small hack that marks the remote "master process" as a local task prior to anything else.

Thanks!

On my remote server..?

This is related to my other thread, so I’ll provide an example there -->

It is. Let me see what else I have set up for MinIO in configs, one moment

Oh nono, more like:

• Create a pipeline
• Add N steps to it
• Run the pipeline
• It fails/succeeds, the user does something with the output
• The user would like to add/modify some steps based on the results now (after closer inspection).I wonder if at (5), do I have to recreate the pipeline every time? 🤔