Any thoughts @<1523701070390366208:profile|CostlyOstrich36> ?
I wouldn’t want to run the entire notebook, just a specific part of it.

@<1537605940121964544:profile|EnthusiasticShrimp49> It’ll take me still some time to find the MVC that generated this, but I do have the ClearML experiment page for it. I was running the thing from ipython , and was trying to create a task from a function:

Oh nono, more like:

• Create a pipeline
• Add N steps to it
• Run the pipeline
• It fails/succeeds, the user does something with the output
• The user would like to add/modify some steps based on the results now (after closer inspection).I wonder if at (5), do I have to recreate the pipeline every time? 🤔

I can't seem to manage the first way around. If I select tasks in different projects, I don't get the bottom bar offering to compare between them

I see that the GUI AutoScaler is only in the paid version, wonder why the GCP driver is not open source?

Heh, well, John wrote that in the first reply in this thread 🙂
And in Task.init main documentation page (nowhere near the code), it says the following -

I'm not sure how the decorators achieve that; from the available examples and trials I've done, it seems that:

• Components anyway need to be available when you define the pipeline controller/decorator, i.e. same codebase
• The component code still needs to be self-composed (or, function component can also be quite complex)
• Decorators do not allow any dynamic build, because you must know how the component are connected at decoration time
  With that said, it could be that the provided example...

We load the endpoint (and S3 credentials) from a .env file, so they're not immediately available at the time of from clearml import Task .
It's a convenience thing, rather than exporting many environment variables that are tied together.

Heh, my bad, the term "user" is very much ingrained in our internal way of working. You can think of it as basically any technically-inclined person in your team or company.

Indeed the options in the WebUI are too limited for our use case, so we're developed "apps" that take a yaml configuration file and build a matching pipeline.
With that, our users do not need to code directly, and we can offer much more fine control over the pipeline.

As for the imports, what I meant is that I encounter...

SuccessfulKoala55 CostlyOstrich36 actually it is the import statement, just finally got around to the traceback:

` File "/home/.../ccmlp/configs/mlops.py", line 4, in <module>
from clearml import Task
File "/home/.../.venv/lib/python3.8/site-packages/clearml/init.py", line 4, in <module>
from .task import Task
File "/home/.../.venv/lib/python3.8/site-packages/clearml/task.py", line 31, in <module>
from .backend_interface.metrics import Metrics
File "/home/......

It can also log generate a log file with this method, it does not have to output it to CONSOLE tab.

I tried that, unfortunately it does not help 😞

Then perhaps mac treats missing environment variables as empty and linux just crashes? Anyway, the config loading should be deferred, shouldn't it?

That could work, given that:
Could we add a preview section? One reason I don't like using the configuration section is that it makes debugging much much harder. Will the clearml-agent download and unzip the files, placing them into the same local folder as needed for execution? What if we want to include non-configuration objects? (i.e. the model case I listed)

But since this has come up a lot recently, any updates on #340? 😍

https://clearml.slack.com/apps/A019MGGA2Q1-around ?

I mean, I see these are defined here https://github.com/allegroai/clearml-agent/blob/master/clearml_agent/definitions.py

But I do not see where an EnvironmentConfig.set() is called...

Am I making sense ?

No, not really. I don't see how task.connect_configuration interacts with our existing CLI? Additionally, the documentation for task.connect_configuration say the second argument is the name of a file, not the path to it? So something is off

I've been answering there as well 🤕

One last MinIO-related question (sorry for the long thread!)
While I do have the access and secret defined in clearml.conf, and even in the WebUI, I still get similar warnings as David does here - https://clearml.slack.com/archives/CTK20V944/p1640135359125200

That's what I found as well, but it did not like it after all (boto is fine with it, but underlying urllib and requests were not?)

It's fine -- I see the added benefit in making sure the users set up their clearml.conf and I've made a script to edit it to our needs as part of the installation process 🙂 Thanks Martin!

AgitatedDove14 another option I thought would be nice is to actually self-sign the internal MinIO bucket, but then I get
[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate (_ssl.c:1076)Are you aware of any other way then (other than the secure: false flag?

~ is a bit weird since it's not part of the package (might as well let the user go through clearml-init ), but using ${PWD} works! 👍 👍

(Though I still had to add the CLEARML_API_HOST and CLEARML_WEB_HOST ofc, or define them in the clearml.conf)

The odd thing is that it was already defined, and then when I clicked an S3 link, it asked me to fill it in again, adding a duplicate credentials row

Sorry to keep this up - what about support for minio using the environment variable? Do I set the CLEARML_FILES_HOST to the end point instead of an s3 bucket?

Great to hear @<1523701087100473344:profile|SuccessfulKoala55> ! Is there an estimated timeline for these releases?

None, they're unusable for us.

I'm not entirely sure I understand the flow but I'll give it a go. I have two final questions:
This seems to only work for a single file (weights_path implies a single file, not multiple ones). Is that the case? Why do you see this as preferred to the dataset method we have now? 🤔

When is the next release expected? 😄

(in the current version, that is, we’d very much like to use them obviously :D)