If nothing specific comes to mind i can try to create some reproducible demo code (after holiday vacation)
Yes please! 🙏
In the mean time see if the workaround is a valid one
How does this work in the context of a pipeline? One of the steps is a multi gpu training that requires accelerate.
We used subprocess for it, ...
Popen? os.system? fork?
It's with decorators.
Interesting, i wasn't aware of this python module for executing accelerate. I'll try to use that.
We used subprocess for it, but for some reason only when invoked in the pipeline the process freezes and doesn't close the main accelerate process. Works fine outside of clearml, any Idea?
If nothing specific comes to mind i can try to create some reproducible demo code (after holiday vacation)
Hi @<1523701949617147904:profile|PricklyRaven28>
Sorry, we missed that one
we need to invoke it with
accelerate launch
so we use
subprocess.run
So you have two options, either you change the script entry of the Task from your " script.py " to" -m accelerate launch script.py
or you manually do that inside your entry point (i.e. call accelerate launch)
BTW, I "think" we added an "auto detect" for it, so that if you launched it manually this way it will know to register it as " -m accelerate launch ... "
How does this work in the context of a pipeline?
Is your pipeline from functions / decorators ? or is it from Tasks ?
(if this is Tasks then just changing the entry point in the overides)
In case of functions or decorators, you have to do that manually (i.e. your function needs to do "accelerate launch"
from accelerate.commands.launch import launch_command, launch_command_parser
parser = launch_command_parser()
args = parser.parse_args("-command -here".split())
launch_command(args)