Would it also be loaded on e.g. Task.running_locally() ?

SuccessfulKoala55 CostlyOstrich36 actually it is the import statement, just finally got around to the traceback:

` File "/home/.../ccmlp/configs/mlops.py", line 4, in <module>
from clearml import Task
File "/home/.../.venv/lib/python3.8/site-packages/clearml/init.py", line 4, in <module>
from .task import Task
File "/home/.../.venv/lib/python3.8/site-packages/clearml/task.py", line 31, in <module>
from .backend_interface.metrics import Metrics
File "/home/......

Hey SuccessfulKoala55 ! Is the configuration file needed for Task.running_locally() ? This is tightly related with issue #395, where we need additional files for remote execution but have no way to attach them to the task other then using the StorageManager as a temporary cache.

That's what I thought too, it should only look for the CLEARML_TASK_ID environment variable?

I see, okay that already clarifies some stuff, I'll dig a bit more into this then! Thanks!

Eek. Is there a way to merge a backup from elastic to current running server?

The screenshot is small since the data is private anyway, but it's enough to see:
"Metric: untitled 00" "plot image" as the image title The attached histogram has a title ("histogram of ...")

I'm aware, but it would be much cleaner to define them in the worker's clearml.conf and let ClearML expose them locally to running tasks.

EDIT: Also the above is specifically about serving, which is not the target here 🤔 At least not yet 😄

We load the endpoint (and S3 credentials) from a .env file, so they're not immediately available at the time of from clearml import Task .
It's a convenience thing, rather than exporting many environment variables that are tied together.

I can also do this via Mongo directly, but I was hoping to skip the K8S interaction there.

At any case, if we were upgrading from e.g. 4.0.0 to 4.1.2, this shouldn't have happened?

I should maybe mention that the security regarding this is low, since this is all behind a private VPN server anyway, I'm mostly just interested in having the credentials used for backtracking purposes

/subscribe
/takemymoney

Let me know if there's any additional information that can help SuccessfulKoala55 !

Full log:
` command: /usr/sbin/helm --version=4.1.2 upgrade -i --reset-values --wait -f=/tmp/tmp77d9ecye.yml clearml clearml/clearml
msg: |-
Failure when executing Helm command. Exited 1.
stdout:
stderr: W0728 09:23:47.076465 2345 warnings.go:70] policy/v1beta1 PodDisruptionBudget is deprecated in v1.21+, unavailable in v1.25+; use policy/v1 PodDisruptionBudget
W0728 09:23:47.126364 2345 warnings.go:70] policy/v1beta1 PodDisruptionBudget is deprecated in v1.21+, unava...

On an unrelated note, when cloning an experiment via the WebUI, shouldn't the cloned experiment have the original experiment as a parent? It seems to be empty

I would expect the service to actually implicitly inject it to new instances prior to applying the user's extra configuration 🤔

https://clearml.slack.com/archives/CTK20V944/p1652350709961419

-ish, still debugging some weird stuff. Sometimes ClearML picks ip and sometimes ip2 , and I can't tell why 🤔

Erm, one sec, let me find that thread...

I am indeed

AgitatedDove14 for future reference this is indeed a PEP-610 related bug, fixed in https://python-poetry.org/blog/announcing-poetry-1.2.0a1/ . I see we can choose the pip version in the config, can we also set the poetry version used? Or is it updated from the lock file itself, or...?

Sure, for example when reporting HTML files:

Does that make sense CostlyOstrich36 ? Any thoughts on how to treat this? For the time being I'm also perfectly happy to include something specific to extra_clearml_conf , but I'm not sure how to set the sdk.aws.s3.credentials to be a list of dictionaries as needed

The error seems to come from this line:
self._driver = _FileStorageDriver(str(path_driver_uri.root)) (line #353 in clearml/storage/helper.py
Where if the path_driver is a local path, then the _FileStorageDriver starts with a base_path = '/' , and then takes extremely long time at iterating over the entire file system (e.g. in _get_objects , line #1931 in helper.py )

Btw TimelyPenguin76 this should also be a good starting point:
First create the target directory and add some files:sudo mkdir /data/clearml sudo chmod 777 -R /data/clearml touch /data/clearml/foo touch /data/clearml/bar touch /data/clearml/bazThen list the files using the StorageManager. It shouldn't take more than a few miliseconds.` from clearml import StorageManager

%%timeit
StorageManager.list("/data/clearml")

-> 21.2 s ± 328 ms per loop (mean ± std. dev. of 7 runs, 1 loop each) `

It's also sufficient to see StorageManager.list("/data/clear") takes a really long time to return no results

TimelyPenguin76 that would have been nice but I'd like to upload files as artifacts (rather than parameters).

AgitatedDove14 I mean like a grouping in the artifact. If I add e.g. foo/bar to my artifact name, it will be uploaded as foo/bar .

proj_suffix = "" i = 2 while Task.get_project_id(f"{proj_name}{proj_suffix}") is not None: tasks = Task.get_tasks(project_name=f"{proj_name}{proj_suffix}") if not [task for task in tasks if not task.get_archived()]: # Empty project, we can use this one... break proj_suffix = f"_{i}" i += 1

Hey FrothyDog40 ! Thanks for clarifying - guess we'll have to wait for that as a feature 😁

Should I create a new issue or just add to this one? https://github.com/allegroai/clearml/issues/529