pipe.add_step(name='stage_process', parents=['stage_data', ],
base_task_project='examples', base_task_name='pipeline step 2 process dataset',
parameter_override={'General/dataset_url': '${stage_data.artifacts.dataset.url}',
'General/test_size': 0.25}, pre_execute_callback=pre_execute_callback_example, post_execute_callback=post_execute_callback_example)

Once defined, the new dataset will have the content of all it's parents. then you can add \ modify \ remove files from it and commit a new dataset.

FiercePenguin76 JitteryCoyote63 are you guys using clearml-serving triton at the moment? If not, would be happy to hear what are the barriers for usage 🙂

WackyRabbit7 pipeline.start_locally() should do the trick I think.

demo code code:
` from clearml.automation import PipelineController

p = PipelineController()

p.start_locally() `
Is that what you had in mind?

Can you also check this:
nslookup app.clear.ml - 8.8.8.8

Hi @<1529271098653282304:profile|WorriedRabbit94> My name is Erez and I'm from ClearML's product team. Thanks for highlighting this! We are aware of the marketplace, and it is in our roadmap to add it there, but no firm date yet.
If you have any comments on adding the plugin to pycharm, or any method that makes it difficult to use, I'm very happy to hear feedback!

If so, what do you mean by "is there a way to reference this file to my process"?

We can't officially confirm nor deny this but yes :sleuth_or_spy:

This gets me the artifact that I return in step1
I think this is what you wanted

@<1541954607595393024:profile|BattyCrocodile47> Thanks a lot for the explanation! These inputs help us a lot building our tools, and eventually, building user's trust in them 🙂 Let us know with what orchestrator you ended up with and how it's going!

Yup indeed! Let us know how it goes!

@<1590514584836378624:profile|AmiableSeaturtle81> Cool to see the community building such things! 🙂 If this works out for you, we'll be happy if you share your process!

A question both to you and @<1541954607595393024:profile|BattyCrocodile47> , what compels you to use a different orchestrator? Anything missing from the ClearML orchestration layer?

Now in step2, I add a pre_execute_callback

That's a good idea! In the autoscaler configuration you can specify an init script, and there you can copy the credentials from s3.

Hmm got it. I think if it's only spin-up time, then it gets more complicated. You need to know that the stopped instance is for this autoscaler, and these instances will need to be manually cleared by users (or they'll continue to pay storage for them, and it's not clear from the autoscaler app that you will be doing that). Do you still think that adding this complexity has merits? What I'm afraid of is hidden costs of stopped instances (Plus, the autoscaler will need internal bookeeping to s...

Yes, any need for anything else?

@<1590514584836378624:profile|AmiableSeaturtle81> yeah I can see what you mean. So you reuploaded everything from the ClearML file server into S3 and just changed the links?

VexedCat68 you mean the artifact in the previous step is called "merged_dataset_id"? Is it an artifact or is it a parameter? And what issues are you having with accessing the parameter?

Hi VexedCat68

Hmm, I'm not 100% sure I follow. you have multiple models doing predictions. Is there a single data source that feeds to them and they run in parallel. or is one's output is another input and they run serially?

Happy to hear 🙂

GiganticTurtle0 That is correct. ATM, you can store some things on the model (I think you can hack it by editing the network configuration and storing whatever you want there.

Just to make sure, if you change the title to "mean top four accuracy" it should work OK

I...Think it's a UI bug? I'll confirm 🙂

Hey, AFAIK, SDK version 1.1.0 disabled the demo server by default (still accessible by setting an envvar).
https://github.com/allegroai/clearml/releases/tag/1.1.0
Is this still an issue even in this version?

Yeah I totally get what you're saying. Basically you want the same code to run locally or remotely, and something external would control whether it runs locally or enqueued to a worker. Am I right?

ExcitedFish86 You came to ClearML because it's free, you stayed because of the magic 🎊 🎉

Yeah I get what you're saying, but when developing ClearML we did not view it like that. when we run it locally or debug it, we thought of it more "This is running just on my local computer without an agent to make sure everything works before I use an agent to run the parts". Sorry that it confuses you 🙂

Hi Moki, Great idea! We'll add it to our plans and update here once it's done 😄

Hmm, can you give a small code snippet of the save code? Are you using a wandb specific code? If so it makes sense we don't save it as we only intercept torch.save() and not wandb function calls