No worries, and I will make sure we output a warning if section names are not used 🙂

GiganticTurtle0 in the PipelineDecorator.component , did you pass helper_functions=[] with refrence to all the sub component ?

MagnificentSeaurchin79 no need for the detection api (yes definitely a mess to setup), it will be more helpful to get a toy example.

DefiantCrab67
Where will you copy it from ?

SubstantialElk6 on the client side?

Agent works when I am running it from virtual environment but stucks in the same place all the time when I using Docker

Can you please provide a log? I'm not sure what it means stuck

WickedElephant66 it should work, how exactly are you calling StorageManager?

Notice that you can embed links to specific view of an experiment, by copying the full address bar when viewing it.

Hi FranticCormorant35
So Tasks have parent field, that would link one to another.
Unfortunately there is no visual representation for it.
What we did with the hyper-parameter for example, was also to add a tag with the ID of the "parent" Task. This would make sense if you have multiple tasks all generated from the same "parent", like in hyper-parameter optimization.
What's your use case ? Is it a single evaluation Task per training, or multiple or con job alike ?

Hmm you mean like overrides ?
Maybe store both before/after resolving ?
(Although that might be confusing? as the before solve should actually be readonly)

Hi SubstantialElk6

We can't seem to find a way for the end user to pass in their git credentials when they run their codes in both agent and non-agent scenarios. Any advice here?

The bottom line is the agent needs to have read-only access to all the repositories so it can launch any Task. I would recommend to create an agent git user with read-only credentials and configure the agent to use it. wdyt?

BTW: see if this works:
$ CLEARML_API_HOST_VERIFY_CERT=0 clearml-init

Hi ReassuredTiger98

However, the clearml-agent also stops working then.

you mean the clearml-agen daemon (the one that spinned the container) is crashing as well ?

Hi DepressedFish57

In my case download each part takes ~5 second, and unzip ~15.

We run into that, and the new version will employ multithreading approach for the unzip (meaning the unzipping will happen in the background)

Okay ConfusedPig65 I found the problem. For some reason the latest TF.keras.load_model . save_model is not tracked.
I'll make sure we push a fix later today

LOL, thanks!

. However, despite having imported the required types from the 

typing

 library in the script where the function decorated with 

PipelineDecorator.component

 is defined, later in the generated script the 

typing

 library is not imported outside the scope of the function

Actually the typing part is not passed to the "created step" , because there are no global imports, for eexample:
` def step(a: pd.DataFrame):
import pandas as pd
...

My plan is to have a AWS Step Functions state machine (DAG) that treats running a ClearML job as one step (task) in the DAG.
...

Yep, that should work
That said, after you have that working, I would actually check pipelines + clearml aws autoscaler, easier setup, and possibly cheaper on the cloud (Lambda vs EC2 instance)

If this works, we might be able to fully replace Metaflow with ClearML!

Can't wait for your blog post on it 😉

So if I am not using remote machine can I disable this?

yes I think you can, add to your clearml.conf
sdk.development.store_jupyter_notebook_artifact = falseBTW: why would you turn it off ?

So what you’re saying is to first kick off a new run and then rename the underlying Pipeline Task, which will cause that particular run to become a new pipeline name?

Correct, basically you are not changing the "pipeline" per-se but the execution name of the pipeline, if that makes sense

What would be most ideal would be to be able to right-click on a pipeline run and have a “clone” option, like you can with a task, where you can start a new run with a new name in a single step.

...

Hi @<1523701949617147904:profile|PricklyRaven28>

I'm trying to figure out if i have a way to report pipeline-step artifact paths in the main pipeline task. (So i don't need to dig into steps to find the artfacts.

Basically this is the monitor_artifacts argument
None

        :param monitor_artifacts: Optional, log the step's artifacts on the pipeline ...

if in the "installed packages" I have all the packages installed from the requirements.txt than I guess I can clone it and use "installed packages"

After the agent finished installing the "requirements.txt" it will put back the entire "pip freeze" into the "installed packages", this means that later we will be able to fully reproduce the working environment, even if packages change (which will eventually happen as we cannot expect everyone to constantly freeze versions)

My problem...

Hi @<1692345677285167104:profile|ThoughtfulKitten41>

Is it possible to trigger a pipeline run via API?

Yes! a pipeline is at the end a Task, you can take the pipeline ID and clone and enqueue it

pipeline_task = Task.clone("pipeline_id_here") 
Task.enqueue(pipeline_task, queue_name="services")

You can also monitor the pipeline with the same Task inyerface.
wdyt?

OutrageousSheep60 so this should work, no?
ds.upload(output_url='gs://<BUCKET>/', compression=0, chunk_size=100000000000)Notice the chunk size is the maximum size (in bytes) per chunk, so it should basically very large

That depends on what you have installed 🙂

but I don’t get to this line, because my task is already of type data_processing

Ohh I see now, it should have added the Tag regardless, you are correct.

I just set

agent.enable_git_ask_pass: true

in the config of the clearml agent (v1.5.1) and the task is still stuck at asking username when trying to get the private dependency.

Hmm that should not happen, could you delete the cache and retry? maybe?

LOL, if this is important we probably could add some support (meaning you will be able to specify it in the "installed packages" section, per Task).
If you find an actual scenario where it is needed, I'll make sure we support it 🙂

BTW: latest PyCharm plugin with 2022 support was just released:
https://github.com/allegroai/clearml-pycharm-plugin/releases/tag/1.1.0

ShortElephant92 yep, this is definitely enterprise feature 🙂
But you can configure user/pass on the open source, even store as hasedh the passwords if you need.