Last but not least - can I cancel the offline zip creation if I'm not interested in it 🤔
EDIT: I see not, guess one has to patch ZipFile ...

We have the following, works fine (we also use internal zip packaging for our models):

model = OutputModel(task=self.task, name=self.job_name, tags=kwargs.get('tags', self.task.get_tags()), framework=framework)
model.connect(task=self.task, name=self.job_name)
model.update_weights(weights_filename=cc_model.save())

@<1523701070390366208:profile|CostlyOstrich36> I added None btw

FWIW running clearml ==1.9.1 with WebApp: 1.9.2-317 • Server: 1.9.2-317 • API: 2.23

Happens with the latest version indeed.
I can’t share our code, but the gist of it is:

pipe = PipelineController(name=..., project=..., version=...)

pipe.add_function_step(...)  # Many calls

pipe.set_default_execution_queue(...)
pipe.start(queue=..., wait=True)

Exactly 🙂

Nothing I can spot --

ClearML results page: 

ClearML pipeline page: 

Launching the next 2 steps
Launching step [...]
Launching step [...]
Launching step: ...
Parameters:
{...}
Configurations:
{}
Overrides:
{}
Launching step: ...
Parameters:
{...}
Configurations:
{}
Overrides:
{}
ClearML Monitor: GPU monitoring failed getting GPU reading, switching off GPU monitoring
2023-02-21 13:53:48
ClearML Monitor: Could not detect iteration reporting, falling back to itera...

I wouldn't put past ClearML automation (a lot of stuff depend on certain suffixes), but I don't think that's the case here hmm

So the pipeline runs successfully, I can find all the different tasks, but I cannot see them in the Pipelines tab…

Thanks SuccessfulKoala55 and AgitatedDove14 ! We'll go through the hoops of setting up mongo on AWS then.
We're working to decouple the data from the helm chart, seems like a dangerous idea to store long term data on k8s in case of failure 😅

@<1523704157695905792:profile|VivaciousBadger56> It seems like whatever you pickled in the zip file relies on some additional files that are not pickled.

Either? 🙂

FWIW It’s also listed in other places @<1523704157695905792:profile|VivaciousBadger56> , e.g. None says:

In order to make sure we also automatically upload the model snapshot (instead of saving its local path), we need to pass a storage location for the model files to be uploaded to.
For example, upload all snapshots to an S3 bucket…

From the one you sent - None

I can only say I’ve found ClearML to be very helpful, even given the documentation issue.
I think they’ve been working on upgrading it for a while, hopefully something new comes out soon.
Maybe @<1523701205467926528:profile|AgitatedDove14> has further info 🙂

Well you could start by setting the output_uri to True in Task.init .

It should store it on the fileserver, perhaps you're missing a configuration option somewhere?

FWIW, we prefer to set it in the agent’s configuration file, then it’s all automatic

We're using self hosted account

But we do use S3

Also I appreciate the time youre taking to answer AgitatedDove14 and CostlyOstrich36 , I know Fridays are not working days in Israel, so thank you 🙂

I am indeed

nevermind! Found and answered (solution in the issue linked above)

Yes; I tried running it both outside venv and inside a venv. No idea why it uses 2.7?

So caching results for steps with the same arguments is trivial. Ultimately I would say you can combine the task-based pipeline with a function-based pipeline to achieve such dynamic control as you specified in the first two scenarios.

About the third scenario I'm not sure. If the configuration has changed, shouldn't the relevant steps (the ones where the configuration changed and their dependent steps) be rerun?

At any case, I think if you stay away from the decorators, at the cost of a bi...

I’ve tracked it down further, it seems the pigar utility does not apply any smart logic there.
The case we have is the following -

• We have a monorepo, but all modules/libs share a common namespace foo ; so e.g. working on module mod , we use from foo.mod import …
• This then looks for a module called foo , even though it’s just a namespace
• In the dist-info requirement, it seems any hyphen, dot, etc are swapped for an underscore, so our site-packages represents this as `foo_m...

minio was a tiny bit of headache to configure, but I'd be happy to help if you want CrookedWalrus33 , I just went through this process yesterday and today (see a few threads up...)

If everything is managed with a git repo, does this also mean PRs will have a messy metadata file attached to them?

Thanks Alon. In the full/official documentation the clearml-data CLI is not mentioned anywhere, so perhaps it should be refreshed 😉

I think we're referring to different things here.
I won't be using the UI (and neither will my team).
But as mentioned, we've used DVC before and it adds a lot of junk metadata files to each GitHub PR (many dvc.yaml , dvc.lock and .gitignore files). We're trying to avoid that as much as possible, hence my question about GitHub pull...

Thanks AgitatedDove14 , I'll first have to prove viability with the free version :)