I should maybe mention that the security regarding this is low, since this is all behind a private VPN server anyway, I'm mostly just interested in having the credentials used for backtracking purposes

Thanks CostlyOstrich36 !
And can I make sure the same budget applies to two different queues?
So that for example, an autoscaler would have a resource budget of 6 instances, and it would listen to aws and default as needed?

We just redeployed to use the 1.1.4 version as Jake suggested, so the logs are gone 😞

I mean, I know I could connect_configuration({k: os.environ.get(k) for k in [...]}) , but then those environment variables would be exposed in the ClearML UI, which is not ideal (the environment variables in question hold usernames and passwords, required for DB access)

Thanks for the reply CostlyOstrich36 !
Does the task read/use the cache_dir directly? It's fine for it to be a cache and then removed from the fileserver; if users want the data to stay they will use the ClearML Dataset 🙂

The S3 solution is bad for us since we have to create a folder for each task (before the task is created), and hope it doesn't get overwritten by the time it executes.

Argument augmentation - say I run my code with python train.py my_config.yaml -e admin.env...

The S3 bucket credentials are defined on the agent, as the bucket is also running locally on the same machine - but I would love for the code to download and apply the file automatically!

QuaintPelican38 did you have a workaround for this then? Some cleanup service or similar?

No it does not show up. The instance spins up and then does nothing.

Maybe it's the missing .bashrc file actually. I'll look into it.

Yup, latest version of ClearML SDK, and we're deployed on AWS using K8s helm

Looks great, looking forward to the all the new treats 😉
Happy new year! 🎉

Would be good if that's mentioned explicitly in the docs 😄 Thanks!

We do not CostlyFox64 , but this is useful for the future 🙂 Thanks!
TimelyPenguin76 I'll have a look, one moment.

Parquet file in this instance (used to be CSV, but that was even larger as everything is stored as a string...)

Maybe create_function_task ? 🤔

Maybe this is part of the paid version, but would be cool if each user (in the web UI) could define their own secrets, and a task could then be assigned to some user and use those secrets during boot?

One must then ask, of course, what to do if e.g. a text refers to a dictionary configuration object? 🤔

Is it currently broken? 🤔

I cannot, the instance is long gone... But it's not different to any other scaled instances, it seems it just took a while to register in ClearML

Note that it would succeed if e.g. run with pytest -s

We have a more complicated case but I'll work around it 😄

Follow up though - can configuration objects refer to one-another internally in ClearML?

I'll have a look, at least it seems to only use from clearml import Task , so unless mlflow changed their SDK, it might still work!

Not that I recall

It's given as the second form you suggested in the mini config ( http://${...}:8080 ). The quotation marks are added later by pyhocon.

This was a long time running since I could not access the macbook in question to debug this.
It is now resolved and indeed a user error - they had implicitly defined CLEARML_CONFIG_FILE to e.g. /home/username/clearml.conf instead of /Users/username/clearml.conf as is expected on Mac.
I guess the error message could be made clearer in this case (i.e. CLEARML_CONFIG_FILE='/home/username/clearml.conf' file does not exist ). Thanks for the support! ❤

Bump SuccessfulKoala55 ?

-ish, still debugging some weird stuff. Sometimes ClearML picks ip and sometimes ip2 , and I can't tell why 🤔

That's what I thought @<1523701087100473344:profile|SuccessfulKoala55> , but the server URL is correct (and WebUI is functional and responsive).
In part of our code, we look for projects with a given name, and pull all tasks in that project. That's the crash point, and it seems to be related to having running tasks in that project.

AgitatedDove14

hmmm... they are important, but only when starting the process. any specific suggestion ?
(and they are deleted after the Task is done, so they are temp)

Ah, then no, sounds temporary. If they're only relevant when starting the process though, I would suggest deleting them immediately when they're no longer needed, and not wait for the end of the task (if possible, of course)

Yeah, and just thinking out loud what I like about the numpy/pandas documentation