Does it make sense to you to run several such glue instances, to manage multiple resource requirements?

TimelyPenguin76 that would have been nice but I'd like to upload files as artifacts (rather than parameters).

AgitatedDove14 I mean like a grouping in the artifact. If I add e.g. foo/bar to my artifact name, it will be uploaded as foo/bar .

Is there a preferred way to stop the agent?

The error seems to come from this line:
self._driver = _FileStorageDriver(str(path_driver_uri.root)) (line #353 in clearml/storage/helper.py
Where if the path_driver is a local path, then the _FileStorageDriver starts with a base_path = '/' , and then takes extremely long time at iterating over the entire file system (e.g. in _get_objects , line #1931 in helper.py )

The S3 bucket credentials are defined on the agent, as the bucket is also running locally on the same machine - but I would love for the code to download and apply the file automatically!

I'm also getting the following warning, I guess it's some ClearML dependency?
IPython could not be loaded!

Any leads TimelyPenguin76 ? I've also tried setting up a minio s3 bucket, but I'm not sure if the remote agent has copied the credentials and host 🤔

It's also sufficient to see StorageManager.list("/data/clear") takes a really long time to return no results

On my remote server..?

This also appears in the error log:

` StorageManager.download_folder(cache_dir.as_posix(), local_folder=".")
File "/home/idan/.clearml/venvs-builds/3.7/lib/python3.7/site-packages/clearml/storage/manager.py", line 278, in download_folder
for path in helper.list(prefix=remote_url):
File "/home/idan/.clearml/venvs-builds/3.7/lib/python3.7/site-packages/clearml/storage/helper.py", line 596, in list
res = self._driver.list_container_objects(self._container, ex_prefix=prefix)
Fi...

some absolute path, i.e. /data/clearml

Sounds like incorrect parsing on ClearML side then, doesn't it? At least, it does not fully support MinIO then

I don't imagine AWS users get a new folder named aws-key-region-xyz-bucket-hostname when they download_folder(...) from an AWS S3 bucket, or do they? 🤔

Ubuntu 18.04, latest clearml version

I... did not, ashamed to admit. The documentation says only boolean values.

So where should I install the latest clearml version? On the client that's running a task, or on the worker machine?

Example configuration -
` version: 1
disable_existing_loggers: true
formatters:
simple:
format: '%(asctime)s %(levelname)-9s %(name)-24s: %(message)s'
filters:
brackets:
(): ccutils.logger.BracketFilter
handlers:
console:
class: ccmlp.utils.TqdmStreamHandler
level: INFO
formatter: simple
filters: [brackets]
loggers: # Set logging levels for specific packages
urllib3:
level: WARNING
matplotlib:
level: WARNING
...

Couldn't the agent just come with the toml library? Kinda easy to load up and check if poetry is present then... 🤔

But yes it indeed used poetry correctly, though it would fail in other circumstances

Hurray! Some success is good 😄

This was changed ~14 months ago, I guess not many people use the local file storage?

https://github.com/allegroai/clearml/commit/95105cbe6a0910ee0c37db707e32549ceaf266eb#diff-9b02810ceabbfd705225d0a7181a53b956b58479e125dd64c5f6c5c8c999e142R373

Eek. Is there a way to merge a backup from elastic to current running server?

The screenshot is small since the data is private anyway, but it's enough to see:
"Metric: untitled 00" "plot image" as the image title The attached histogram has a title ("histogram of ...")

That gives us the benefit of creating "local datasets" (confined to the scope of the project, do not appear in Datasets tabs, but appear as normal tasks within the project)

That's probably in the newer ClearML server pages then, I'll have to wait still 😅

I just ran into this too recently. Are you passing these also in the extra_clearml_conf for the autoscaler?

FWIW It’s also listed in other places @<1523704157695905792:profile|VivaciousBadger56> , e.g. None says:

In order to make sure we also automatically upload the model snapshot (instead of saving its local path), we need to pass a storage location for the model files to be uploaded to.
For example, upload all snapshots to an S3 bucket…

Not sure I understand your comment - why not let the user start with an empty comparison page and add them from "Add Experiment" button as well?

Yes exactly, but I guess I could've googled for that 😅

Copy the uncommitted changes captured by ClearML using the UI, write to changes.patch , run git apply changes.patch 👍

I'm trying, let's see; our infra person is away on holidays :X Thanks! Uh, which configuration exactly would you like to see? We're running using the helm charts on K8s, so I don't think I have direct access to the agent configuration/update it separately?

From the log you shared, the task is picked up by the 

worker_d1bd92a3b039400cbafc60a7a5b1e52b_4e831c4cbaf64e02925b918e9a3a1cf6_<hostname>:gpu0,1

 worker

I can try and target the default one if it helps..?

Ah I see, if the pipeline controller begins in a Task it does not add the tags to it…