Loading part from task B:
` def get_models_from_task(task: clearml.Task, model_artifact_substring: str = 'iter_') -> dict:
"""
Extract all models saved as artifacts with the specified substring

:param task: Task to fetch from
:param model_artifact_substring: Substring for recognizing models among artifacts
:return: Mapping between iter number and model instance
"""

# Extract models from task (models are named iter-XXX where XXX is the iteration number)
model_...

I need code access, not UI access

Couldn't find any logic on which tasks fail and why... all the lines are exactly the same, only different parameters

I'm asking that because the DSes we have are working on multiple projects, and they have only one trains.conf file, I wouldn't want them to edit it each time they switch project

Okay so that is a bit complicated

In our setup, the DSes don't really care about agents, the agents are being managed by our MLops team.
So essentially if you imagine it the use case looks like that:
A data scientists wants to execute some CPU heavy task. The MLops team supplied him with a queue name, and the data scientist knows that when he needs something heavy he pushes it there - the DS doesn't know nothing about where it is executed, the execution environment is fully managed by the ML...

I double checked that the task object is the correct one

I want to get the instances of the tasks executed by this controller task

CostlyOstrich36

Any news on this? This is kind of creepy, it's something so basic that I can't trust my prediction pipeline because sometimes it fails randomly with no reason

How do I get all children tasks given a parent?

Another thing I noticed now it happens on my personal computer, when I execute the same pipeline from the exact same commit with exact same data on another host it works without these problems

this is another execution:

inference table is a pandas dataframe

I re-executed the experiemnt, nothing changes

TimelyPenguin76 if I build a custom image, do I have to host it on dockerhub for it to run on the agent? If not how do I make the agent aware of my custom image?

SuccessfulKoala55 The simplest thing i can think of is on Task.execute_remotely to be able to append ot the docker_init_bash_script

to fix it, I excluded this var entirely from the docker-compose

I dont think that has to do anything with the value zero, the lines that should come out of 'mean' and 'median' have the value of None under quantile, but have a dre_0.5 assoxiated with them. those lines appear in the notebook and not in the ui

yep

Committing that notebook with changes solved it, but I wonder why it failed

yep, just sec

Please update me 🙂

I only have like 40 tasks including the example ones

192.168.1.71?

so it is not defined

got it

SuccessfulKoala55

Yes, I have the controller task in hand