It's not exactly "debugging", but rather a description of the generated model/framework (generated with pygraphviz).

I think ClearML boots up only afterwards, so those environment variables may not be available yet.
You should set them manually in the bootstrap code unfortuantely.

Actually TimelyPenguin76 I get only the following as a "preview" -- I thought the preview for an image would be... the image itself..?

Created - https://github.com/allegroai/clearml/issues/709

… And it’s failing on typing hints for functions passed in pipe.add_function_step(…, helper_function=[…]) … I guess those aren’t being removed like the wrapped function step?

Seems like you're missing an image definition (AMI or otherwise)

Ultimately we're trying to avoid docker in AWS autoscaler (virtualization on top of virtualization seems redundant), and instead we maintain an AMI for a faster boot sequence.
We had no issues when we used pip , but now when trying to work with poetry all these issues came up.
The way I understand poetry to work, is that it is expected there is one system-wide installation that is used for virtual environment creation and manipulation. So at least it may be desired that the ...

Thanks SuccessfulKoala55 , I made https://github.com/allegroai/clearml-agent/issues/126 as a suggestion.

Do you have any thoughts on how to expose these... manually?
It does so already for environment variables that prefixed with CLEARML_ , so it would be nice to have some control over that.

I mean, I see these are defined here https://github.com/allegroai/clearml-agent/blob/master/clearml_agent/definitions.py

But I do not see where an EnvironmentConfig.set() is called...

SweetBadger76 TimelyPenguin76
We're finally tackling this (since it has kept us back at 1.3.2 even though 1.6.2 is out...), and noticed that now the bucket name is also part of the folder?

So following up from David's latest example:
StorageManager.download_folder(remote_url='s3://****-bucket/david/', local_folder='./')Actually creates a new folder ./****-bucket/david/ and puts it contents there.

EDIT: This is with us using internal MinIO, so I believe ClearML parses that end...

Scaling to zero, copying the mongodb data, and scaling back up worked like a charm.

Thanks @<1523701827080556544:profile|JuicyFox94> !

I'll have some reports tomorrow I hope TimelyPenguin76 SuccessfulKoala55 !

Anything specific we should look into TimelyPenguin76 ?

Well, -ish. Ideally what we're after is one of the following:
Couple a task with a dataset. Keep it visible in it's destined location. Create a dataset separately from the task. Have control over its visibility and location. If it's hidden, it should not affect normal UI interaction (most annoying is having to click twice on the same project name when there are hidden datasets, which do not appear in the project view)

Or to be clear, the environment installed by the autoscaler under /clearml_agent_venv has poetry installed, and it uses that to set up the environment for the executed task, e.g. in root/.clearml/venvs-builds/3.10/task_repository/.../.venv , but the latter does not have poetry installed, and so it crashes?

We're using the example autoscaler, nothing modified

I'm not entirely sure I understand the flow but I'll give it a go. I have two final questions:
This seems to only work for a single file (weights_path implies a single file, not multiple ones). Is that the case? Why do you see this as preferred to the dataset method we have now? 🤔

Ah I see, if the pipeline controller begins in a Task it does not add the tags to it…

Okay so the new functionality is maintained; thanks!

We're still working these quirks out. But one issue after we changed the AMI is that the VPC (SubnetId?) was missing from the instance so it could not reach the ClearML API server.

I think maybe the autoscaler service is missing some additional settings...

Any sneak preview? 😉 😁

Without knowing anything, I'm assuming maybe ClearML patches plt.title and not Axes.set_title ?

Thanks, that's what I thought - so I'm missing something else in the installation. I'll dig further 🙂

So a missing bit of information that I see I forgot to mention, is that we named our packages as foo-mod in pyproject.toml . That hyphen then get’s rewritten as foo_mod.x.y.z-distinfo .

foo-mod @ git+

Added the following line under volumes for apiserver , fileserver , agent-services :
- /data/clearml:/data/clearml

The screenshot is small since the data is private anyway, but it's enough to see:
"Metric: untitled 00" "plot image" as the image title The attached histogram has a title ("histogram of ...")

Oh! Nice! I'll have a go at it and report back at the PR if it's in a functional state 🙂 Thanks AgitatedDove14 !

Adding bucket = clearml in aws.s3.credentials did not help either

some absolute path, i.e. /data/clearml

Am I making sense ?

No, not really. I don't see how task.connect_configuration interacts with our existing CLI? Additionally, the documentation for task.connect_configuration say the second argument is the name of a file, not the path to it? So something is off