Exactly; the cloud instances (that are run with clearml-agent ) should have that clearml.conf + any changes specified in extra_clearml_configuration for the scaler

I think you're looking for the execute_remotely function?

Yes exactly 👍 Good news.

Always great to find a bug! I'll make relevant SDK updates then.

I am indeed

Here's a full description of the layout:
Remote agent + entire ClearML docker suite running on host A. Host A also has a /data/clearml folder mounted to it and to it's docker containers (I've edited the docker-compose to add this mount point) Connect to host A, use StorageManager on the /data/clearml folder for some early troubles (e.g. long .list call) Use the same connection to run a task with execute_remotely and download_folder and see it crash :disapp...

So a normal config file with environment variables.

Actually TimelyPenguin76 I get only the following as a "preview" -- I thought the preview for an image would be... the image itself..?

Or some users that update their poetry.lock and some that update manually as they prefer to resolve on their own.

Is it CLEARML_CONFIG_FILE ? (I had to dig this from the GH code 😅 )

This also appears in the error log:

` StorageManager.download_folder(cache_dir.as_posix(), local_folder=".")
File "/home/idan/.clearml/venvs-builds/3.7/lib/python3.7/site-packages/clearml/storage/manager.py", line 278, in download_folder
for path in helper.list(prefix=remote_url):
File "/home/idan/.clearml/venvs-builds/3.7/lib/python3.7/site-packages/clearml/storage/helper.py", line 596, in list
res = self._driver.list_container_objects(self._container, ex_prefix=prefix)
Fi...

Will try!

Curious - is there a temporary changelog for 1.2.0? 😁 Always fun to poke at the upcoming features

EDIT: Wait, should the clearml RC be installed outside the venv for the agent as well?

Yes

That's fine as well - the code simply shows the name of the environment variable, not it's value, since that's taken directly from the agent listening to the services queue (and who's then running the scaler)

That's what I thought too, it should only look for the CLEARML_TASK_ID environment variable?

Couldn't the agent just come with the toml library? Kinda easy to load up and check if poetry is present then... 🤔

But yes it indeed used poetry correctly, though it would fail in other circumstances

I have selected the correct project indeed

Kinda, yes, and this has changed with 1.8.1.
The thing is that afaik currently ClearML does not officially support a remotely executed task to spawn more tasks, so we also have a small hack that marks the remote "master process" as a local task prior to anything else.

The logs are on the bucket, yes.
The default file server is also set to s3://ip:9000/clearml

Sure! It looks like this

… And it’s failing on typing hints for functions passed in pipe.add_function_step(…, helper_function=[…]) … I guess those aren’t being removed like the wrapped function step?

CostlyOstrich36 That looks promising, but I don't see any documentation on the returned schema (i.e. workers.worker_stats is not specified anywhere?)

Right so it uses whatever version is available on the agent.
Yeah it would be nice to have either a poetry_version (a-la https://github.com/allegroai/clearml-agent/blob/5afb604e3d53d3f09dd6de81fe0a494dacb2e94d/docs/clearml.conf#L62 ), rename the latter to manager_version , or just install from the captured environment, etc? 🤔

Could also be related to K8, so pinging JuicyFox94 just in case 😉

Not sure if ClearML has any built in support, but we used the above for a similar issue but with Prefect2 :)

So a missing bit of information that I see I forgot to mention, is that we named our packages as foo-mod in pyproject.toml . That hyphen then get’s rewritten as foo_mod.x.y.z-distinfo .

foo-mod @ git+

Either one would be nice to have. I kinda like the instant search option, but could live with an ENTER to search.

I opened this meanwhile - https://github.com/allegroai/clearml-server/issues/138

Generally, it would also be good if the pop-up presented some hints about what went wrong with fetching the experiments. Here, I know the pattern is incomplete and invalid. A less advanced user might not understand what's up.

Thanks!

I realized it might work too, but looking for a more definitive answer 😄 Has no-one attempted this? 🤔

Okay, I'll test it out by trying to downgrade to 4.0.0 and then upgrade to 4.1.2
Just to make sure, the chart_ref is allegroai/clearml right? (for some reason we had clearml/clearml and it seems like it previously worked?)