Sure thing :)

If the manual execution (i.e. pycharm) was working it should have stored it on the Pipeline Task.

task = Task.init(project_name='debug', task_name='test tqdm cr cl') print('start') for i in tqdm.tqdm(range(100), dynamic_ncols=True,): sleep(1) print('done')This code snippet works as expected (console will show the progress at the flush interval without values in between). What's the difference ?!

Thanks for checking NastyFox63
I double checked with both front/backend , there should not be any limit...
Could you maybe provide a toy demo to reproduce the issue ?

I think you have it on the workers and queues page when you click on the worker you have its detials

Those variables are not passed to the remote instance they are used by the aws autoscaler to launch it, but there is no need to pass them.
I think the easiest is to add them to the "extra_vm_bash_script" as well

Hmm let me check first when it is going to upgraded and if there is a workaround

I can't think of any hack that will satisfy your IT other than than an actual vault...
wdyt?

yes you are correct, OS environment:
TRAINS_PROC_MASTER_ID=1:task_id_here

Nice!

script, and the kwcoco not imported directly (but from within another package).

fyi: usually the assumption is that clearml will only list the directly imported packages, as these will pull the respective required packages when the agent will be installing them ... (meaning that if in the repository you are never actually directly importing kwcoco, it will not be listed (the package that you do import directly, the you mentioned is importing kwcoco, will be listed). I hope this ...

Correct

SweetGiraffe8
That might be it, could you test with the Demo server ?

Done

Ohh, like a query based only of the stored url ?
Do you also has the creating Task ?

Many thanks!

Correct 🙂

If you one each "main" process as a single experiment, just don't call Task.init in the scheduler

I will TIAS, but maybe worthwhile to also mention if it has to be the absolute path or if relative path is fine too!

Good point! (absolute but you can use ~, and I "think" also $ENV )

But this is not copy, this is mount, your log showed cp failing

Hi TrickySheep9
You should probably check the new https://github.com/allegroai/clearml-server-helm-cloud-ready helm chart 😉
https://github.com/allegroai/clearml-server-helm-cloud-ready

Hmm @<1523701083040387072:profile|UnevenDolphin73> I think this is the reason, None
and this means that even without a full lock file poetry can still build an environment

You should have a download button when you hover over the table, I guess that would be the easiest.
If needed I can send an SDK code but unfortunately there is no single call for that

Hi @<1523701304709353472:profile|OddShrimp85>
You mean something like clearml-serving ?
None

Having the ability to pack jobs/tasks onto the same "resource" (underlying server/EC2 instance)

This is essentially a "queue". Basically a queue is a way to abstract a specific type of resource, so that you can achieve exactly what you descibed.

open up a streaming use case, wherein batch (offline) inference could be done directly inside of a ClearML pipeline in reaction to an event/trigger (like new data landing in your data lake).

Yes, that's exactly how clearml is designed, a...

Wow, thank you very much. And how would I bind my code to task?

you mean the code that creates pipeline Tasks ?
(remember the pipeline itself is a Task in the system, basically if your pipeline code is a single script it will pack the entire thing )

Maybe the configuration file changed?
None
The logic is if the name and project are the same, and there are no artifacts/models, and the last time it was created was under 72 hours, reuse the Task

So if any step corresponding to 'inference_orchestrator_1' fails, then 'inference_orchestrator_2' keeps running.

GiganticTurtle0 I'm not sure it makes sense to halt the entire pipeline if one step fails.
That said, how about using the post_execution callback, then check if the step failed, you could stop the entire pipeline (and any running steps), what do you think?

Hey SarcasticSparrow10 see here 🙂
https://allegro.ai/clearml/docs/docs/deploying_clearml/clearml_server_linux_mac.html#upgrading

so if the node went down and then some other node came up, the data is lost

That might be the case. where is the k8s running ? cloud service ?

If you want to quickly test it:
pip install clearml-agent
Then assuming Task id is aabbcc
Run
clearml-agent execute --id aabbccYou should be able to trace if the package was installed