Unbelievable! That worked.

@<1523701083040387072:profile|UnevenDolphin73> : A big point for me is to reuse/cache those artifacts/datasets/models that need to be passed between the steps, but have been produced by colleagues' executions at some earlier point. So for example, let the pipeline be A(a) -> B(b) -> C(c), where A,B,C are steps and their code, excluding configurations/parameters, and a,b,c are the configurations/parameters. Then I might have the situation, that my colleague ran the pipeline A(a) -> B(b) -> C(c...

What is helper_functions in

• None
• Nonegood for?
  I do not find any example. The descriptions says

By default the pipeline step function has no access to any of the other functions, by specifying additional functions here, the remote pipel...

My entire code is

from clearml import Task, TaskTypes
task = Task.init(project_name='FirstTrial', task_name='first_trial', task_type=TaskTypes.training)
PACKAGE_VERSION = '0.4.1'
dataset_name = "Demodata"

which I - now - also ran as a whole script.

But still, in the web app the task is considered to be still "running". I am not sure what to do, so that the task is considered to be "completed".

Maybe this is only logged after it is not "running" anymore, but I am not sure how to "complete" a task programmatically.

But then I do not see the "installed packages" in "excecution".

I am using a venv environment, managed by poetry. Not sure if that helps.

KindChimpanzee37 : Thank you so much! I asked follow up questions 🙂 .

KindChimpanzee37 , I ensured that the dataset_name is the same in get_data.py and preprocessing.py and that seemed to help. Then, I got the error RuntimeError: No audio I/O backend is available. , because of which I installed PySoundFile with pip; that helped. Weirdly enough then, the old id error came back. So, I re-ran get_data.py and then preprocessing.py - this time the id error was gone again. Instead, I got `raise TypeError("Invalid file: {0!r}".format(self.name))
TypeError:...

"uncommitted changes" and "container" is also empty.

But, I guess @<1523701070390366208:profile|CostlyOstrich36> wrote that in a different chat, right?

@<1523701205467926528:profile|AgitatedDove14> Is it true that, when using the "pipeline from tasks" approach, my Python environment in which the pipeline is programmed, does not need to know any of the code with which the tasks have been programmed and still the respective pipeline would be executed just fine?

Also, I could not find any larger examples on github about Model, InputModel, or OutputModel. It's kind of difficult to build a PoC this way... 😅

GrittyStarfish67 : Thanks! But how are those for ClearML vs MLRun? Granted, ClearML has a ~5 times more github stars than MLRun, but besides that: Both are from mid 2019 according to releases on git. I have not been in their slack and I know nothing about community adoption. (Btw, Kedro has twice as many stars than ClearML - even if it has far fewer feature, those that it does have, seem pretty well done.)

@<1523701070390366208:profile|CostlyOstrich36> : After more playing around, it seems that ClearML Server does not store the models or artifacts itself. These are stored somewhere else (e.g., AWS S3-bucket) or on my local machine and ClearML Server is only storing configuration parameters and previews (e.g., when the artifact is a pandas dataframe). Is that right? Is there a way to save the models completely on the ClearML server?

@<1523701205467926528:profile|AgitatedDove14> : In general: If I do not build a package out of my local repository/project , I cannot reference anything
from the local project/repository directly, right? I must make a package out of it, or I must reference it with the repo argument, or I must reference respective functions using the helper_functions argument. Did I get this right?

@<1523701205467926528:profile|AgitatedDove14>

I have... its included in None

No these are 3 different ways of building pipelines.

That is what I meant to say 🙂 , sorry for the confusion, @<1523701205467926528:profile|AgitatedDove14> .

@<1523701083040387072:profile|UnevenDolphin73> , your point is a strong one. What are clear situations in which pipelines can only be build from tasks, and not one of the other ways? An idea would be if the tasks are created from all kinds of - kind of - unrelated projects where the code that describes the pipeline does not ...

@<1523701205467926528:profile|AgitatedDove14> In the documentation it warns about .close() "Only call Task.close if you are certain the Task is not needed."
What does the documentation refer to? My understanding would be that if close the task within a program, I am not able to use the task object anymore as before and I need to retrieve it via query_tasks to get it again. Is that correct?

I think now that here the documentation means the usage of connect.

Understand. Here I only see:

>pip show clearml
WARNING: Ignoring invalid distribution -upyterlab (c:\users\...\lib\site-packages)
WARNING: Ignoring invalid distribution -illow (c:\users\...\lib\site-packages)
Name: clearml
Version: 1.6.4
Summary: ClearML - Auto-Magical Experiment Manager, Version Control, and MLOps for AI
Home-page: None
`Auth...

AgitatedDove14 : Not sure: They also have the feature store (data management), as mentioned, which is pretty MLOps-y 🙂 . Also, they do have workflows ( https://docs.mlrun.org/en/latest/concepts/multi-stage-workflows.html ) and artifacts/model management ( https://docs.mlrun.org/en/latest/store/artifacts.html ) and serving ( https://docs.mlrun.org/en/latest/serving/serving-graph.html ).

@<1523701070390366208:profile|CostlyOstrich36> : Thanks, where can I find more information on ClearML's model repository. I hardly find any in the documentation.

Also, that leaves the question open, what Model is for. I described how I understand, the workflow should look like, but my question remains open...

GrittyStarfish67 : In terms of "has a good name" you literally mean the name or do you mean, they have a good reputation 😄 ?

@<1523701070390366208:profile|CostlyOstrich36> indeed

Do you mean "exactly" as in "you finally got it" or in the sense of "yes, that was easy to miss"?

It is documented at None ... super deep in the code. If you don't know that output_uri in TASK's (!) init is relevant, you would never know...