Ahh. This is a shame. I really want to use ClearML to efficiently compute features but it’s proving a challenge!
Thanks

You may indeed 😊 thanks!

from tempfile import mkdtemp new_folder = with_feature.get_mutable_local_copy(mkdtemp())It’s this line that causes the issue

Ah ok. I’m guessing the state file is auto uploaded in the background? I haven’t kicked that off “intentionally”

ClearML shows the new (child) dataset as “Uploading”, if that helps…

Do notice this will only work for pipelines from Tasks, is this a good fit for you?

The issue with this is that we would then presumably have to run/“build” each of the Tasks (pipeline steps) separately to put them on the ClearML server and then get their Task ID’s in order to even write the code for the Pipeline, which increases the complexity of any automated CI/CD flow. Correct me if I’m wrong.

Essentially, I think the key thing here is we want to be able to build the entire Pipe...

Thanks, I’ll check out those GitHub Actions examples but as you say, it’s the “template” step that is the key bit for this particular application.

the pipeline from tasks serializes itself to a configuration object that you can edit/create externally

I think if it has to come down to fiddling with lower-level objects, I’ll hold off for now and wait until something a bit more user-friendly comes along. Depends on how easy this is to work with.

This is something that we do need if we a...

The Pipeline is defined using PipelineDecorators, so currently to “build and run” it would just involve running the script it is defined in (which enqueues it and runs it etc).

This is not ideal, as I need to access the Task ID and the only methods I can see are for use within the Task/Pipeline ( Task.current_task and PipelineDecorator.get_current_pipeline )

The reason I want to check completion etc outside the Pipeline Task is that I want to run data validation etc once when the pipe...

The pseudo-code you wrote previously is what would be required, I believe

be able to get the pipeline’s Task ID back at the end

This is the missing piece. We can’t perform validation without this, afaik

Sorry, I think something’s got lost in translation here, but thanks for the explanation.

Hopefully this is clearer:

• Say we have a new ClearML pipeline as code on a new commit in our repo.
• We want to build and run this new pipeline and have it available on the ClearML Server.
• We want to run a suite of tests that validate/verify/etc the performance of this entire ClearML Pipeline, e.g. by having it run on a set of predefined inputs and checking the various artifacts that were creat...

The issue here is I don’t have the pipeline ID as it is a new version of the pipeline - i.e. the code has been updated, I want to run the updated pipeline (for the first time), get its ID, and then analyse the run/perform updates to tags (for example)

Sorry, I don’t understand how this helps with validating the pipeline run.
Where would the validation code sit?
And the ClearML Pipeline run needs to be available on the ClearML Server (at least as a draft) so that it can be marked as in-production and cloned in the future

Yep, would be happy to run locally, but want to automate this so does running locally help with getting the pipeline run ID (programmatically)?

For reference, this works as expected:

e.g. pseudo for illustration only
` def get_list(dataset_id):
from clearml import Dataset
ds= Dataset.get(dataset_id=dataset_id)
ds_dir=ds.get_local_copy()
etc...
return list_of_objs # one for each file, for example

def pipeline(dataset_id):
list_of_obj = get_list(dataset_id)
list_of_results = []
for obj in list_of_obj:
list_of_results.append(step(obj))
combine(list_of_results) `One benefit is being able to make use of the Pipeline caching so if ne...

I get an error about incorrect Task ID’s – in the above pseudo code it would be the ID of the step Task that was displayed in the error

Hi John, we are using a self-hosted server with:

WebApp 1.9.2-317
Server: 1.9.2-317
API: 2.23

edit: clearml==1.11.0

Yes, sorry, the final cell has the flush followed by the close

I have already tested that the for loop does work, including caching, when spinning out multiple Tasks.

As I say, the issue is grouping the results of the tasks into a list and passing them into another step

Hmm. Okay. Thanks

Ahh okay.

I’m an absolute numpty.

I had enabled caching on the Pipeline Task that was grabbing a load of ClearML IDs and so it was trying to “get” datasets that had since been deleted.

Thanks for the nudge to minimal test – silly I didn’t do it before asking!

Appreciate your help.

I used task.flush(wait_for_uploads=True) in the final cell of the notebook

I have added a lot of detail to this, sorry.

The inline comments in the code talk about that specific script/implementation.

I have added a lot of context in the doc string at the top.

(including caching, even if the number of elements in the list of vals changes)

So the DAG is getting confused on bringing the results of the Tasks together

Is there a rule whereby only python native datatypes can be used as the “outer” variable?

I have a dict of numpy np.array s elsewhere in my code and that works fine with caching.

I basically just mean having a date input like you would in excel where it brings up a calendar and a clock if it’s time – and defaults to “now”

my colleague, @<1534706830800850944:profile|ZealousCoyote89> has been looking at this – I think he has used the relevant kwarg in the component decorator to specify the packages, and I think it worked but I’m not 100%. Connah?

The return objects were stored to S3 but PipelineDecorator.upload_artifact still uploaded to the file server. Not sure what was up with that but as explained in my next comment it did work when I tried again.

It also seems that PipelineDecorator.upload_artifact is not compatible with caching, sadly, but that is another issue for another thread that I will be starting on Monday.

Have a good weekend

Producing it now — thanks for your help, won’t be a few mins