I mean, I know I could connect_configuration({k: os.environ.get(k) for k in [...]}) , but then those environment variables would be exposed in the ClearML UI, which is not ideal (the environment variables in question hold usernames and passwords, required for DB access)

Any simple ways around this for now? @<1523701070390366208:profile|CostlyOstrich36>

One last MinIO-related question (sorry for the long thread!)
While I do have the access and secret defined in clearml.conf, and even in the WebUI, I still get similar warnings as David does here - https://clearml.slack.com/archives/CTK20V944/p1640135359125200

If I add the bucket to that (so CLEARML_FILES_HOST= s3://minio_ip:9000/minio/bucket ), I then get the following error instead --

2021-12-21 22:14:55,518 - clearml.storage - ERROR - Failed uploading: SSL validation failed for ... [SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:1076)

Maybe it's the missing .bashrc file actually. I'll look into it.

Because I never know which repo to open it in 😁

It is

I'm running tests with pytest , it consumes/owns the stream

This seems to be fine for now, if any future lookups finds this thread, btw
with mock.patch('clearml.datasets.dataset.Dataset.create'): ...

Tada! https://github.com/allegroai/clearml-agent/issues/90

FYI @<1523701087100473344:profile|SuccessfulKoala55> (or I might be doing something wrong), but it seems the python migration code comes with carriage returns, so it fails on linux by default (one has to tr -d '\r' to use it)

EDIT: And also it defaults to /opt/allegro/data rather than the recommended /opt/clearml/data which is suggested when installing the server 🤔

Thanks SuccessfulKoala55 ! Could I change this during runtime, so for example, only the very first task goes through this process?

Great to hear @<1523701087100473344:profile|SuccessfulKoala55> ! Is there an estimated timeline for these releases?

I didn't mention code in #340 nor did I mention data here 😄 The idea was to package non git-specific files for remote execution

TimelyPenguin76 I added pip install --update clearml-agent to the extra_vm_bash_script for the autoscaler, that should at least guarantee the latest clearml agent is used on the instance, right?

Then the username and password would be visible in the autoscaler task 😕
But it should work out of the box, as it does work like that out of the box also regardless of ClearML. The user and personal access token are used as is and it propagates down to submodules, since those are simply another git repository.
I've further checks on a different machine and it works as well 🤔

They are set with a .env file - it's a common practice. The .env file is, at the moment, uploaded to a temporary cache (if you remember the discussion regarding the StorageManager ), so it's also available remotely (related to issue #395)

So the config loading is not deferred until execution 😞

That could work, given that:
Could we add a preview section? One reason I don't like using the configuration section is that it makes debugging much much harder. Will the clearml-agent download and unzip the files, placing them into the same local folder as needed for execution? What if we want to include non-configuration objects? (i.e. the model case I listed)

Yes -- that's what I meant by The title is specified in the plot . I make the plots manually - title, axes labels, ticks, etc. In that sense, the figure is entirely configured. ClearML just saves it as "untitled 00/plot image"

For example, we have a complicated YAML file with built-in !include instructions, so we upload all the included files too. This then clogs up the artifacts sidebar, and it would be nice to be able to say "these are all artifacts from this one file, you can collapse it by clicking here"

I'll kill the agent and try again but with the detached mode 🤔

These issues popped when I upgraded to clearml-agent 1.5.0.

That could be a solution for the regex search; my comment on the pop-up (in the previous reply) was a bit more generic - just that it should potentially include some information on what failed while fetching experiments 😄

I also tried switching to dockerized mode now, getting the same issue 🤔

First bullet point - yes, exactly

Second bullet point - all of it, really. The SDK documentation and the examples.
For example, the Task object is heavily overloaded and its documentation would benefit from being separated into logical units of work. It would also make it easier for the ClearML team to spot any formatting issues.
Any linked example to github is welcome, but some visualization/inline code with explanation is also very much welcome.

The overall flow I currently have is e.g.
Start an internal task (not ClearML Task; MLOps not initialized yet) Call some pre_init function with args so I can upload the environment file via StorageManager to S3 Call some start_run function with the configuration dictionary loaded, so I can upload the relevant CSV files and configuration file Finally initialize the MLOps (ClearML), start a task, execute remotely
I can play around with 3/4 (so e.g. upload CSVs and configuratio...

BTW AgitatedDove14 following this discussion I ended up doing the regex way myself to sync these, so our code has something like the following. We abuse the object description here to store the desired file path.

` config_path = task.connect_configuration(configuration=config_path, name=config_fname)
included_files = find_included_files_in_source(config_path)
while included_files:
file_to_include = included_files.pop()
sub_config = task.connect_configuration(
configurat...

StaleButterfly40 what use case are you looking for? I've used environment variables in the config file and then I can overwrite them in os.environ before ClearML loads the config

You mean at the container level or at clearml?

Yes, the container level (when these docker shell scripts run).
The per user ID would be nice, except I upload the .env file before the Task is created (it's only available really early in the code).