Yes, as I wrote above 😄

So a normal config file with environment variables.

Could you provide a more complete set of instructions, for the less inclined?
How would I backup the data in future times etc?

• in the second scenario, I might have not changed the results of the step, but my refactoring changed the speed considerably and this is something I measure.
• in the third scenario, I might have not changed the results of the step and my refactoring just cleaned the code, but besides that, nothing substantially was changed. Thus I do not want a rerun.Well, I would say then that in the second scenario it’s just rerunning the pipeline, and in the third it’s not running it at all 😄
  (I ...

Yeah I will probably end up archiving them for the time being (or deleting if possible?).

Otherwise (regarding the code question), I think it’s better if we continue the original thread, as it has a sample code snippet to illustrate what I’m trying to do.

Yeah I was basically trying to avoid clutter in the Pipelines page. But see my other thread for the background, maybe you have some good input there? 🙏

See e None @<1523701087100473344:profile|SuccessfulKoala55>

Dynamic pipelines in a notebook, so I don’t have to recreate a pipeline every time a step is changed 🤔

Hey @<1523701435869433856:profile|SmugDolphin23> , thanks for the reply! I’m aware of the caching — that’s not the issue I’m trying to resolve 🙂

This is related to my other thread, so I’ll provide an example there -->

@<1523701827080556544:profile|JuicyFox94> we have it up and running, hurray 🙂
One thing I noticed in the k8s logs is frequent warnings about Python 3.6..? Is the helm chart built with that Python version?

/usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: Python 3.6 is no longer supported by the Python core team. Therefore, support for it is deprecated in cryptography and will be removed in a future release.
from cryptography.utils import int_...

And actually it fails on quite many tasks for us with this Python 3.6.
I tried to set up a different image ( agent8sglue.defaultContainerImage: "ubuntu:20.04" ) but that did not change much.
I suspect the culprit is agentk8sglue.image , which is set to tag 1.24-21 of clearml-agent-k8s-base . That image is quite very old… Any updates on that? 🤔

Thanks!

Thanks! To clarify, all the agent does is then spawn new nodes to cover the tasks?

i.e. It does not process tasks on its own?

I am; it seems like maybe a couple of hours?

Thanks CostlyOstrich36 !

Not that I recall

The deferred_init input argument to Task.init is bool by default, so checking type(deferred_init) == int makes no sense to begin with, and is altering the flow.

Thanks AgitatedDove14 , I'll first have to prove viability with the free version :)

I believe it is maybe a race condition that's tangent to clearml now...

Because I never know which repo to open it in 😁

Illustrating the first bullet point

SuccessfulKoala55 This happens pip >= 22.3 btw.

Another semi-related issue is that I now encounter these kind of error messages:
clearml_agent: ERROR: __init__() got an unexpected keyword argument 'types'

My current approach with pipelines basically looks like a GH CICD yaml config btw, so I give the user a lot of control on which steps to run, why, and how, and the default simply caches all results so as to minimize the number of reruns.

The user can then override and choose exactly what to do (or not do).

From the traceback ( backend_interface/task/task.py, line 178, in __init__ ), notice it's not Task.init

I'm not sure why internally ClearML tries to initialize a task when get_task is called...

proj_suffix = "" i = 2 while Task.get_project_id(f"{proj_name}{proj_suffix}") is not None: tasks = Task.get_tasks(project_name=f"{proj_name}{proj_suffix}") if not [task for task in tasks if not task.get_archived()]: # Empty project, we can use this one... break proj_suffix = f"_{i}" i += 1

It's a small snippet that ensures identically named projects are still unique'd with a running number.

From the one you sent - None