Oh nono, more like:

• Create a pipeline
• Add N steps to it
• Run the pipeline
• It fails/succeeds, the user does something with the output
• The user would like to add/modify some steps based on the results now (after closer inspection).I wonder if at (5), do I have to recreate the pipeline every time? 🤔

Yeps

PricklyRaven28 That would be my fallback, it would make development much slower (having to build containers with every small change)

Literally just as you said it - it happened

The results from searching in the "Add Experiment" view (can't resize column widths -> can't see project name ...)

Any updates @<1523701087100473344:profile|SuccessfulKoala55> ? 🙂

Hm, I'm not sure I follow 🤔 How does the API server config relate to the file server?

And this is of course strictly with the update to 1.6.3 (or newer) that should support API 2.20

From our IT dept:

Not really, when you launch the instance, the launch has to already be in the right VPC/Subnet. Configuration tools are irrelevant here.

I believe it is maybe a race condition that's tangent to clearml now...

No worries @<1537605940121964544:profile|EnthusiasticShrimp49> ! I made some headway by using Task.create , writing a temporary Python script, and using task.update in a similar way to how pipeline steps are created.

I'll try and create an MVC to reproduce the issue, though I may have strayed from your original suggestion because I need to be able to use classes and not just functions.

Uhhh, but pyproject.toml does not necessarily entail poetry... It's a new Python standard

Now, the original pyhocon does support include statements as you mentioned - https://github.com/chimpler/pyhocon

Oh! Nice! I'll have a go at it and report back at the PR if it's in a functional state 🙂 Thanks AgitatedDove14 !

Done https://github.com/allegroai/clearml-docs/issues/294

So the ..data referenced in the example above are part of the git repository?
What about setting the working_directory to the user working directory using Task.init or Task.create ?

I mean, I know I could connect_configuration({k: os.environ.get(k) for k in [...]}) , but then those environment variables would be exposed in the ClearML UI, which is not ideal (the environment variables in question hold usernames and passwords, required for DB access)

Any simple ways around this for now? @<1523701070390366208:profile|CostlyOstrich36>

One last MinIO-related question (sorry for the long thread!)
While I do have the access and secret defined in clearml.conf, and even in the WebUI, I still get similar warnings as David does here - https://clearml.slack.com/archives/CTK20V944/p1640135359125200

If I add the bucket to that (so CLEARML_FILES_HOST= s3://minio_ip:9000/minio/bucket ), I then get the following error instead --

2021-12-21 22:14:55,518 - clearml.storage - ERROR - Failed uploading: SSL validation failed for ... [SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:1076)

SweetBadger76 TimelyPenguin76
We're finally tackling this (since it has kept us back at 1.3.2 even though 1.6.2 is out...), and noticed that now the bucket name is also part of the folder?

So following up from David's latest example:
StorageManager.download_folder(remote_url='s3://****-bucket/david/', local_folder='./')Actually creates a new folder ./****-bucket/david/ and puts it contents there.

EDIT: This is with us using internal MinIO, so I believe ClearML parses that end...

I'm saying it's a bug

Maybe it's the missing .bashrc file actually. I'll look into it.

Because I never know which repo to open it in 😁

It is

I'm running tests with pytest , it consumes/owns the stream

This seems to be fine for now, if any future lookups finds this thread, btw
with mock.patch('clearml.datasets.dataset.Dataset.create'): ...

Tada! https://github.com/allegroai/clearml-agent/issues/90

FYI @<1523701087100473344:profile|SuccessfulKoala55> (or I might be doing something wrong), but it seems the python migration code comes with carriage returns, so it fails on linux by default (one has to tr -d '\r' to use it)

EDIT: And also it defaults to /opt/allegro/data rather than the recommended /opt/clearml/data which is suggested when installing the server 🤔

Thanks SuccessfulKoala55 ! Could I change this during runtime, so for example, only the very first task goes through this process?