They are set with a .env file - it's a common practice. The .env file is, at the moment, uploaded to a temporary cache (if you remember the discussion regarding the StorageManager ), so it's also available remotely (related to issue #395)

Thanks @<1537605940121964544:profile|EnthusiasticShrimp49> ! That’s definitely the route I was hoping to go, but the create_function_task is still a bit of a mystery, as I’d like to use an entire class with relevant logic and proper serialization for inputs, and potentially I’ll need to add more “helper functions” (as in the case of DataTransformationStep , for example). Any thoughts on that? 🤔

No need for S3 or other resources?

Oh nono, more like:

• Create a pipeline
• Add N steps to it
• Run the pipeline
• It fails/succeeds, the user does something with the output
• The user would like to add/modify some steps based on the results now (after closer inspection).I wonder if at (5), do I have to recreate the pipeline every time? 🤔

Thanks for your help SuccessfulKoala55 ! Appreciate the patience 🙏

Great, thanks! Any idea about environment variables and/or other files (CSV)? I suppose I could use the task.upload_artifact for the CSVs. but I'm still unsure about the environment variables

Maybe it's the missing .bashrc file actually. I'll look into it.

Any thoughts SuccessfulKoala55 ?

Let me verify a hypothesis...

Nice, that works 🙂 Thanks!

But there's nothing of that sort happening. The process where it's failing is on getting tasks for a project.

It failed on some missing files in my remote_execution, but otherwise seems fine now

I'm trying, let's see; our infra person is away on holidays :X Thanks! Uh, which configuration exactly would you like to see? We're running using the helm charts on K8s, so I don't think I have direct access to the agent configuration/update it separately?

Latest (1.5.1 I believe?), full log incoming, but it's like I've posted elsewhere already 🤔
It just sets up the environment and immediately crashes when trying to run the code.
The setup itself is done correctly.

Hey FrothyDog40 ! Thanks for clarifying - guess we'll have to wait for that as a feature 😁

Should I create a new issue or just add to this one? https://github.com/allegroai/clearml/issues/529

So basically I'm wondering if it's possible to add some kind of small hierarchy in the artifacts, be it sections, groupings, tabs, folders, whatever.

So kind of the ability to have more artifact types in "Artifacts" tab, other than Other and OutputModels , etc

The SDK is fine as it is - I'm more looking at the WebUI at this point

For example, we have a complicated YAML file with built-in !include instructions, so we upload all the included files too. This then clogs up the artifacts sidebar, and it would be nice to be able to say "these are all artifacts from this one file, you can collapse it by clicking here"

Opened this - https://github.com/allegroai/clearml/issues/530 let me know if it's not clear enough FrothyDog40 !

TimelyPenguin76 that would have been nice but I'd like to upload files as artifacts (rather than parameters).

AgitatedDove14 I mean like a grouping in the artifact. If I add e.g. foo/bar to my artifact name, it will be uploaded as foo/bar .

Still the same issue it seems (or maybe the behaviour is the new behaviour?)

That's what I thought @<1523701087100473344:profile|SuccessfulKoala55> , but the server URL is correct (and WebUI is functional and responsive).
In part of our code, we look for projects with a given name, and pull all tasks in that project. That's the crash point, and it seems to be related to having running tasks in that project.

Heh, good @<1523704157695905792:profile|VivaciousBadger56> 😁
I was just repeating what @<1523701070390366208:profile|CostlyOstrich36> suggested, credits to him

One last MinIO-related question (sorry for the long thread!)
While I do have the access and secret defined in clearml.conf, and even in the WebUI, I still get similar warnings as David does here - https://clearml.slack.com/archives/CTK20V944/p1640135359125200

Well you could start by setting the output_uri to True in Task.init .

SuccessfulKoala55 could this be related to the monkey patching for logging platform? We have our own logging handlers that we use in this case

But... Which queue does it listen to, and which type of instances will it use etc

Hey @<1523701070390366208:profile|CostlyOstrich36> , thanks for the reply!
I’m familiar with the above repo, we have the ClearML Server and such deployed on K8s.
What’s lacking is documentation regarding the clearml-agent helm chart. What exactly does it offer, etc.
We’re interested in e.g. using karpenter to scale our deployments per demand, effectively replacing the AWS autoscaler.

We’re using karpenter (more magic keywords for me), so my understanding is that that will manage the scaling part.