Yeah, and just thinking out loud what I like about the numpy/pandas documentation

Also full disclosure - I'm not part of the ClearML team and have only recently started using pipelines myself, so all of the above is just learnings from my own trials 😅

We just do task.close() and then start a new task.Init() manually, so our "pipelines" are self-controlled

Why not give ClearML read-only access credentials to the repository?

Ah, you meant “free python code” in that sense. Sure, I see that. The repo arguments also exist for functions though.

Sorry for hijacking your thread @<1523704157695905792:profile|VivaciousBadger56>

Internally yes, but in Task.init the default argument is a boolean, not an int.
We don't want to close the task, but we have a remote task that spawns more tasks. With this change, subsequent calls to Task.init fail because it goes in the deferred init clause and fails on validate_defaults .

Kinda, yes, and this has changed with 1.8.1.
The thing is that afaik currently ClearML does not officially support a remotely executed task to spawn more tasks, so we also have a small hack that marks the remote "master process" as a local task prior to anything else.

SmugDolphin23 I think you can simply change not (type(deferred_init) == int and deferred_init == 0) to deferred_init is True ?

Oh, well, no, but for us that would be one way solution (we didn't need to close the task before that update)

So now we need to pass Task.init(deferred_init=0) because the default Task.init(deferred_init=False) is wrong

But it is strictly that if condition in Task.init, see the issue I opened about it

1.8.3; what about when calling task.close() ? We suddenly have a need to setup our logging after every task.close() call

True, and we plan to migrate to pipelines once we have some time for it :) but anyway that condition is flawed I believe

I narrowed the bug down to the "fix" in 1.8.1, see my other post

First bullet point - yes, exactly

Second bullet point - all of it, really. The SDK documentation and the examples.
For example, the Task object is heavily overloaded and its documentation would benefit from being separated into logical units of work. It would also make it easier for the ClearML team to spot any formatting issues.
Any linked example to github is welcome, but some visualization/inline code with explanation is also very much welcome.

But we do use S3

I've updated my feature request to describe that as well. A textual description is not necessarily a preview 😅 For now I'll use the debug samples.

These kind of things definitely show how ClearML was designed originally only for neural networks tbh, where images are almost always only part of the dataset. Same goes for the consistent use of iteration everywhere 😞

SuccessfulKoala55 The changelog wrongly cites https://github.com/allegroai/clearml/issues/400 btw. It is not implemented and is not related to being able to save CSVs 😅

If everything is managed with a git repo, does this also mean PRs will have a messy metadata file attached to them?

I guess it depends on what you'd like to configure.
Since we let the user choose parents, component name, etc - we cannot use the decorators. We also infer required packages at runtime (the autodetection based on import statements fails with a non-trivial namespace) and need to set that to all components, so the decorators do not work for us.

There's code that strips the type hints from the component function, just think it should be applied to the helper functions too :)

Woot! What about clearml-agent 1.2.0? 🙂

Also, creating from functions allows dynamic pipeline creation without requiring the tasks to pre-exist in ClearML, which is IMO the strongest point to make about it

Nice, that works 🙂 Thanks!

Would be great if it is 😍 We have few files that change frequently and are quite large in size, and it would be quite a storage hit to save all of them

I'll have a look, at least it seems to only use from clearml import Task , so unless mlflow changed their SDK, it might still work!

Oh! Nice! I'll have a go at it and report back at the PR if it's in a functional state 🙂 Thanks AgitatedDove14 !

Is it currently broken? 🤔

Thanks LazyFish41 , that explains it 👍

Hey @<1537605940121964544:profile|EnthusiasticShrimp49> ! You’re mostly correct. The Step classes will be predefined (of course developers are encouraged to add/modify as needed), but as in the DataTransformationStep , there may be user-defined functions specified. That’s not a problem though, I can provide these functions with the helper_functions argument.

• The .add_function_step is indeed a failing point. I can’t really create a task from the notebook because calling `Ta...