I mean those, that you see in the screen shot. The difference in code is - at least for me - to write

• parameters_data = {'custom1': 'no', 'custom2': False}; parameters_data = task.connect(parameters_data , name='data')
• task.set_user_properties(custom1='no', custom2=False)
  

KindChimpanzee37 : Thank you so much! I asked follow up questions 🙂 .

Secondly, I do not understand this:

None says

Manually mark a Task as completed. This will close the running process and will change the Task’s status to Completed (Use this function to close and change status of remotely executed tasks). To simply change the Task’s status to completed, use task.close()

None says

Closes the current Task and cha...

I created None

@<1523701205467926528:profile|AgitatedDove14> : Wait, so, if a task is initialized in process A and I call mark_completed in a process B, which process is terminated? A or B?

I am running it in the Python Console in PyCharm with Task.init. I get with the log:

ClearML Task: overwriting (reusing) task id=dfa2dff538d54c18ad97ea1593cbd357
2023-02-14 13:06:44,336 - clearml.Task - WARNING - Failed auto-detecting task repository: [WinError 123] Die Syntax für den Dateinamen, Verzeichnisnamen oder die Datenträgerbezeichnung ist falsch: '[...]\<input>'
ClearML results page: [None](https://app.clear.ml/projects/9acc061c880344a881790461a4baa837/experiments/dfa2dff538d54c1...

@<1523701083040387072:profile|UnevenDolphin73> : I see. I did not make the connection that output_uri=True is what I was missing. I thought this was the default. But the default is actually "None", which is different than "True".

It is documented at None ... super deep in the code. If you don't know that output_uri in TASK's (!) init is relevant, you would never know...

@<1523701083040387072:profile|UnevenDolphin73> : If I do, what should I configure how?

Unbelievable! That worked.

I have already been trying to contribute (have three pull requests), but honestly I feel it is a bit weird, that I need to update a documentation about something I do not understand, while I actually try to evaluate if ClearML is the right tool for our company...

@<1523701205467926528:profile|AgitatedDove14> , you wrote

• Components anyway need to be available when you define the pipeline controller/decorator, i.e. same codebase

No you an specify a different code base, see here:

Is the code in this "other" repo downloaded to the agent's machine? Or is the component's code pushed to the machine on which the repository is?
If the second case is true: How is the other machine (on which the other repo is lying on) turned into an agent?

@<1523701205467926528:profile|AgitatedDove14> Is it true that, when using the "pipeline from tasks" approach, my Python environment in which the pipeline is programmed, does not need to know any of the code with which the tasks have been programmed and still the respective pipeline would be executed just fine?

@<1523701083040387072:profile|UnevenDolphin73> : I do not see any way to download the model manually from the web app either. All I see is the link to the file on my harddrive (see shreenshot).

The second process says there is not file at all. I think, all that happened is that the update_weights only uploaded the location of the .zip file (which we denote as a .model file) on my harddrive, but not the file itself.
![image](https://clearml-web-assets.s3.amazonaws.com/scoold/image...

@<1523701205467926528:profile|AgitatedDove14> : I am writing quite a bit of documentation on the topic of pipelines. I am happy to share the article here, once my questions are answered and we can make a pull request for the official documentation out of it.

@<1523701087100473344:profile|SuccessfulKoala55> : I referenced this conversation in the issue None

I am not sure if it the fact the name of the file ends with .model is an issue - but that would be somewhat crazy design...

The first scenario is you standard "the code stays the same, the configuration changes" for the second step. Here, I want
The second and third scenario is "the configuration stays the same, the code changes", this is the case, e.g., if code is refactored, but effectively does the same as before.

@<1523701083040387072:profile|UnevenDolphin73> , you wrote

About the third scenario I'm not sure. If the configuration has changed, shouldn't the relevant steps (the ones where the configuration...

@<1523701087100473344:profile|SuccessfulKoala55> I think I might have made a mistake earlier - but not in the code I posted before. Now, I have the following situation:

• In my training Python process on my notebook I train the custom made model and put it on my harddrive as a zip file. Then I run the code

output_model = OutputModel(task=task, config_dict={...}, name=f"...")
output_model.update_weights(weights_filename=r"C:\path\to\mymodel.zip", is_package=True)

2. I delete the "...

@<1523701087100473344:profile|SuccessfulKoala55> Also, I think that - in this case, but also in other cases - the issue is not just the documentation, but also the design of the SDK.

@<1523701083040387072:profile|UnevenDolphin73> , you wrote

Well, I would say then that in the second scenario it’s just rerunning the pipeline, and in the third it’s not running it at all

(I mean, in both the code may have changed, the only difference is that in the second case you’re interested in measuring it, and in the third,
you’re not, so it sounds like a user-specific decision).

Well, I would hope that in the second scenario step A is not rerun. Yes, in the third scen...

@<1523701070390366208:profile|CostlyOstrich36> , I am build a PoC, evaluating if we should use ClearML for our entire ML team and go Scale or Enterprise pricing. For that I need to know all/most capabilities and concepts of ClearML to see if ClearML is future-proof.

TL;DR: difficult to narrow it down, but we (amongst other things), we need a model store

What is helper_functions in

• None
• Nonegood for?
  I do not find any example. The descriptions says

By default the pipeline step function has no access to any of the other functions, by specifying additional functions here, the remote pipel...

"using your method you may not reach the best set of hyperparameters."

Of course you are right. It is an efficiency trade-off of speed vs effectiveness. Whether this is worth it or not depends on the use-case. Here it is worth it, because the performance of the modelling is not sensitive to the parameter we search for first. Being in the ball-park is enough. And, for the second set of parameters, we need to do a full grid search (the parameters are booleans and strings); thus, this wo...

@<1523701083040387072:profile|UnevenDolphin73> : No, I love it ❤ . Now, I just have to read everything 😄 .

@<1523701205467926528:profile|AgitatedDove14> : In general: If I do not build a package out of my local repository/project , I cannot reference anything
from the local project/repository directly, right? I must make a package out of it, or I must reference it with the repo argument, or I must reference respective functions using the helper_functions argument. Did I get this right?

@<1523701070390366208:profile|CostlyOstrich36>

My training outputs a model as a zip file. The way I save and load the zip file to make up my model is custom made (no library is directly used), because we invented the entire modelling ourselves. What I did so far:

output_model = OutputModel(task=..., config_dict={...}, name=f"...")
output_model.update_weights("C:\io__path\...", is_package=True)

and I am trying to load the model in a different Python process with

mymodel =...

@<1523701083040387072:profile|UnevenDolphin73> : A big point for me is to reuse/cache those artifacts/datasets/models that need to be passed between the steps, but have been produced by colleagues' executions at some earlier point. So for example, let the pipeline be A(a) -> B(b) -> C(c), where A,B,C are steps and their code, excluding configurations/parameters, and a,b,c are the configurations/parameters. Then I might have the situation, that my colleague ran the pipeline A(a) -> B(b) -> C(c...

🙂

Last point on component caching, what I suggest is actually providing users the ability to control the cache "function". Right now (a bit simplified but probably accurate), this is equivalent to hashing of the following dict:

{"code": "code here", "container": "docker image", "container args": "docker args", "hyper-parameters": "key/value"}

We could allow users to add a function that get's this dict and returns a new dict that will be used for hashing. This way we will e...