I will try checking the PipelineController documentation for that.
Thank you @<1523701070390366208:profile|CostlyOstrich36> 🙂

It's a bit of a problem to do this, as I'm using a subprocess to run a python script in the container, and the paths in my local differ from the one inside the container.

Thanks @<1523701070390366208:profile|CostlyOstrich36> !
I'll try to have a look on this type of implementation 🙂

I can see from the console in the UI that a part of the command it's trying to run is:
'echo \'Binary::apt::APT::Keep-Downloaded-Packages "true";\' > /etc/apt/apt.conf.d/docker-clean'

and some more commands that I'm trying to understand why does my agent gets it. I'm going back and forth on the clearml config but everything I change doesn't seem to have any effect.

Thank you @<1523701435869433856:profile|SmugDolphin23> ! 🙏
I'll try it out

Thanks @<1523701070390366208:profile|CostlyOstrich36> !
so, both steps are running on the same machine. I would love to get your suggestion in this case.

yes!
I'll try that out. Thank you @<1523701070390366208:profile|CostlyOstrich36>

Thank you @<1523701070390366208:profile|CostlyOstrich36> and @<1523701087100473344:profile|SuccessfulKoala55> !
I managed to get what I wanted using your inputs!

Thanks @<1576381444509405184:profile|ManiacalLizard2>
for now it seems to work just fine, as you said

@<1523701070390366208:profile|CostlyOstrich36>
Yes!
But, I want it to use the streaming output of the parent task to trigger start of the child task (it will be in the middle of the run, as opposed to regular pipeline where it launches the next task automatically after the parent task is done).

Why is it not possible to use argparse and then when I do:
Task.init() I add:

auto_connect_arg_parser=False

and then I read the arguments to dict and use task.connect(arg_dict, name=my_custom_name)?

I tried doing that but it doesn't work when I clone a task from the UI (meaning it will not work in pipeline as well)

Thank you @<1523701087100473344:profile|SuccessfulKoala55> !

and in general - how many agents can I open on a machine with 16 CPUs and no GPUs?

Hi @<1523701070390366208:profile|CostlyOstrich36> , thanks for the response!
I want to implement the user arguments for the task using argparse, but I want to change the default "Args" name to a name that I will set

Hi @<1523701087100473344:profile|SuccessfulKoala55> ,
First - I initiate the agent using this command:
clearml-agent daemon --queue maytar_test_q --docker docker_image --detached --cpu-only

As for the task itself - I have a bash script inside the container that executes a python script (also located inside the container), that is getting arguments via argparse (so far, no clearml involved). To initiate the task - I run a very basic python script (out of the container) that initiates a ...

@<1523701070390366208:profile|CostlyOstrich36>
You're right, I do use a custom entry point in my docker file.
So, can you please suggest if you think this would work:

• Set an environment that will be able to run this task entirely (script will include Task.init).
• Create a new image from which I will delete the customised run command (FYI that the Dockerfile does not contain clearml/clearml-agent installation)
• Run the task from python script - will publish task to clearml UI
• Clone task...