Hi all, I'm training a model using AWS sagemaker and monitoring with a clearML server on-prem. Works well enough when the training is split (horovod - with a task on each rank). But when I try and spawn eval jobs to run on different AWS machines, it seems that the Task.init kills the job.
Note that spawning the eval jobs work fine without clearML.
I'm a little out of my depth figuring out whats wrong, can anyone tell what I am missing?
Thanks!
Edit: in case it is relevant: the local training script creates the first task and the rest are in the cloud while the eval jobs are spawned from the training jobs so they all start in the cloud. That is the most significant difference I can think of between the training jobs that work fine and the eval jobs that fail.
Hi IrateDolphin19 ,
Can you give a bit of a simplistic schema of what you're doing or trying to achieve? Are you using pipelines for this?
IrateDolphin19 , can you give a bit of an explanation on how and what you're doing, and what on the clearml side seems to fail - how do you create the tasks and manage them...
not sure if this makes it more or less clear 😕
I'm not familiar with pipelines, I don't believe I'm using it
