KindChimpanzee37 : Ok, will do. (More question from my side though. :-D) But I need to have pretty good idea before presenting our concept to the bosses.

KindChimpanzee37 : Thank you so much! I asked follow up questions 🙂 .

@<1523701083040387072:profile|UnevenDolphin73> : From which URL is your most recent screenshot?

What is helper_functions in

• None
• Nonegood for?
  I do not find any example. The descriptions says

By default the pipeline step function has no access to any of the other functions, by specifying additional functions here, the remote pipel...

@<1523701205467926528:profile|AgitatedDove14> : In general: If I do not build a package out of my local repository/project , I cannot reference anything
from the local project/repository directly, right? I must make a package out of it, or I must reference it with the repo argument, or I must reference respective functions using the helper_functions argument. Did I get this right?

@<1523701083040387072:profile|UnevenDolphin73> : I do not see any way to download the model manually from the web app either. All I see is the link to the file on my harddrive (see shreenshot).

The second process says there is not file at all. I think, all that happened is that the update_weights only uploaded the location of the .zip file (which we denote as a .model file) on my harddrive, but not the file itself.
![image](https://clearml-web-assets.s3.amazonaws.com/scoold/image...

@<1523701083040387072:profile|UnevenDolphin73> : I see. I did not make the connection that output_uri=True is what I was missing. I thought this was the default. But the default is actually "None", which is different than "True".

But, I guess @<1523701070390366208:profile|CostlyOstrich36> wrote that in a different chat, right?

The first scenario is you standard "the code stays the same, the configuration changes" for the second step. Here, I want
The second and third scenario is "the configuration stays the same, the code changes", this is the case, e.g., if code is refactored, but effectively does the same as before.

@<1523701083040387072:profile|UnevenDolphin73> , you wrote

About the third scenario I'm not sure. If the configuration has changed, shouldn't the relevant steps (the ones where the configuration...

@<1523701205467926528:profile|AgitatedDove14> : "does that make sense ?" Not really.

"you do not need to automatically Add/Log/Track things into the Task in the current process." - I do not need to automatically do [...]? You mean I can do it automatically, but alternatively I can do it manually? Do you mean I use close within a process to prevent automatic logging/adding/tracking? But, as far as I know, after I used close I am not able to log etc. manually either. So...

"Mark...

@<1523701083040387072:profile|UnevenDolphin73> : How do you figure? In the past, my colleagues and I just shared the .zip file via email / MS Teams and it worked. So I don't think so.

@<1523701083040387072:profile|UnevenDolphin73> : A big point for me is to reuse/cache those artifacts/datasets/models that need to be passed between the steps, but have been produced by colleagues' executions at some earlier point. So for example, let the pipeline be A(a) -> B(b) -> C(c), where A,B,C are steps and their code, excluding configurations/parameters, and a,b,c are the configurations/parameters. Then I might have the situation, that my colleague ran the pipeline A(a) -> B(b) -> C(c...

No these are 3 different ways of building pipelines.

That is what I meant to say 🙂 , sorry for the confusion, @<1523701205467926528:profile|AgitatedDove14> .

@<1523701083040387072:profile|UnevenDolphin73> , your point is a strong one. What are clear situations in which pipelines can only be build from tasks, and not one of the other ways? An idea would be if the tasks are created from all kinds of - kind of - unrelated projects where the code that describes the pipeline does not ...

@<1523701070390366208:profile|CostlyOstrich36> : After more playing around, it seems that ClearML Server does not store the models or artifacts itself. These are stored somewhere else (e.g., AWS S3-bucket) or on my local machine and ClearML Server is only storing configuration parameters and previews (e.g., when the artifact is a pandas dataframe). Is that right? Is there a way to save the models completely on the ClearML server?

@<1523701070390366208:profile|CostlyOstrich36> indeed

I just see the website that I linked to. I am not sure what is meant by "python environment". I cannot make a screen shot, because I do not know where to look for this in the first place.

But still, in the web app the task is considered to be still "running". I am not sure what to do, so that the task is considered to be "completed".

>pip show clearml
WARNING: Ignoring invalid distribution -upyterlab (c:\users\...\lib\site-packages)
WARNING: Ignoring invalid distribution -illow (c:\users\...\lib\site-packages)
Name: clearml
Version: 1.6.4
Summary: ClearML - Auto-Magical Experiment Manager, Version Control, and MLOps for AI
Home-page: None
`Auth...

My entire code is

from clearml import Task, TaskTypes
task = Task.init(project_name='FirstTrial', task_name='first_trial', task_type=TaskTypes.training)
PACKAGE_VERSION = '0.4.1'
dataset_name = "Demodata"

which I - now - also ran as a whole script.

Ah... if I run the same script not from PyCharm, but from the terminal, then it gets completed... puh...

I see that with task.mark_completed(), I am able to programmatically complete the task.

@<1523701087100473344:profile|SuccessfulKoala55> I think I might have made a mistake earlier - but not in the code I posted before. Now, I have the following situation:

• In my training Python process on my notebook I train the custom made model and put it on my harddrive as a zip file. Then I run the code

output_model = OutputModel(task=task, config_dict={...}, name=f"...")
output_model.update_weights(weights_filename=r"C:\path\to\mymodel.zip", is_package=True)

2. I delete the "...

@<1523701083040387072:profile|UnevenDolphin73> : If I do, what should I configure how?

Understand. Here I only see:

@<1523701205467926528:profile|AgitatedDove14> : I am writing quite a bit of documentation on the topic of pipelines. I am happy to share the article here, once my questions are answered and we can make a pull request for the official documentation out of it.

KindChimpanzee37 , any idea 🙂 ?

Here is my code:

from clearml import Task, TaskTypes
from clearml.task_parameters import TaskParameters, param, percent_param

class MyParams(TaskParameters):

    iterations = param(
        type=int,
        desc="Number of iterations to run",
        range=(0, 100000),
    )

    target_accuracy = percent_param(
        desc="The target accuracy of the model",
    )

myPar = MyParams(iterations=1000, target_accuracy=0.95)
parameters_to_track1 = {'var1': 'a', 'hyper_par': 1}
parameter...

AgitatedDove14 : Not sure: They also have the feature store (data management), as mentioned, which is pretty MLOps-y 🙂 . Also, they do have workflows ( https://docs.mlrun.org/en/latest/concepts/multi-stage-workflows.html ) and artifacts/model management ( https://docs.mlrun.org/en/latest/store/artifacts.html ) and serving ( https://docs.mlrun.org/en/latest/serving/serving-graph.html ).

@<1537605940121964544:profile|EnthusiasticShrimp49> : The biggest advantage I see to split your code into pipeline components is caching. A little bit structuring your code, but I was told by the staff this should not one's main aim with ClearML components. What is your main take away for splitting your code into components?

My HPO on top of the pipeline is already working 🙂 I am currently experimenting on using the HPO in a (other) pipeline that creates two HPO steps (from the same funct...

Thank you I found the error.
myPar = task.connect(myPar, name='from TaskParameters')
is required.