That was a good idea, unfortunately did not help too much, but I think I may have a found a work around, thanks!

Aw you deleted your response fast CostlyOstrich36 xD

Indeed it does not appear in ps aux so I cannot simply kill it (or at least, find it).
I was wondering if it's maybe just a zombie in the server API or similar

It's removed now, magic of asking for help and doing nothing 😄

Literally just as you said it - it happened

Thanks!

Created None

We're using the example autoscaler, nothing modified

We're using self hosted account

Nothing I can spot --

ClearML results page: 

ClearML pipeline page: 

Launching the next 2 steps
Launching step [...]
Launching step [...]
Launching step: ...
Parameters:
{...}
Configurations:
{}
Overrides:
{}
Launching step: ...
Parameters:
{...}
Configurations:
{}
Overrides:
{}
ClearML Monitor: GPU monitoring failed getting GPU reading, switching off GPU monitoring
2023-02-21 13:53:48
ClearML Monitor: Could not detect iteration reporting, falling back to itera...

@<1523701070390366208:profile|CostlyOstrich36> I added None btw

I believe that a Pipeline should have the system tags ( pipeline , maybe hidden ), even if it created in a running Task .

Happens with the latest version indeed.
I can’t share our code, but the gist of it is:

pipe = PipelineController(name=..., project=..., version=...)

pipe.add_function_step(...)  # Many calls

pipe.set_default_execution_queue(...)
pipe.start(queue=..., wait=True)

So the pipeline runs successfully, I can find all the different tasks, but I cannot see them in the Pipelines tab…

FWIW running clearml ==1.9.1 with WebApp: 1.9.2-317 • Server: 1.9.2-317 • API: 2.23

When I use the APIClient to fetch the tags for the project, I get an empty collection of system tags:

<projects.GetProjectTagsResponse: {
    "tags": [],
    "system_tags": []
}>

Ah I see, if the pipeline controller begins in a Task it does not add the tags to it…

Yes, exactly. I have not yet had a chance to try this out -- should it work?

I... did not, ashamed to admit. The documentation says only boolean values.

No, I have no running agents listening to that queue. It's as if it's retained in some memory somewhere and the server keeps creating it.

Could also be related to K8, so pinging JuicyFox94 just in case 😉

I can only say I’ve found ClearML to be very helpful, even given the documentation issue.
I think they’ve been working on upgrading it for a while, hopefully something new comes out soon.
Maybe @<1523701205467926528:profile|AgitatedDove14> has further info 🙂

Looks great, looking forward to the all the new treats 😉
Happy new year! 🎉

I guess it does not do so for all settings, but only those that come from Session()

@<1523704157695905792:profile|VivaciousBadger56> It seems like whatever you pickled in the zip file relies on some additional files that are not pickled.

Samoja ongelmia täälläkin 😅

… And it’s failing on typing hints for functions passed in pipe.add_function_step(…, helper_function=[…]) … I guess those aren’t being removed like the wrapped function step?

Heh, well, John wrote that in the first reply in this thread 🙂
And in Task.init main documentation page (nowhere near the code), it says the following -

I have no idea what’s the difference, but it does not log the internal repository 😞 If I knew why, I would be able to solve it myself… hehe

I wouldn't put past ClearML automation (a lot of stuff depend on certain suffixes), but I don't think that's the case here hmm

FWIW It’s also listed in other places @<1523704157695905792:profile|VivaciousBadger56> , e.g. None says:

In order to make sure we also automatically upload the model snapshot (instead of saving its local path), we need to pass a storage location for the model files to be uploaded to.
For example, upload all snapshots to an S3 bucket…