@<1523701070390366208:profile|CostlyOstrich36> can you help me?

@<1523701070390366208:profile|CostlyOstrich36> Can you help me with my question?

@<1523701435869433856:profile|SmugDolphin23> I added os.environ["NCCL_SOCKET_IFNAME" and I managed to run on nccl
But it seems that workaround that you said do not run 2 processes on 2 nodes, but 4 processes on 4 different nodes
current_conf = task.launch_multi_node(args.nodes*args.gpus)
os.environ["NODE_RANK"] = str(current_conf.get("node_rank", ""))
os.environ["NODE_RANK"] = str(current_conf["node_rank"] // args.gpus)
`os.environ["LOCAL_RANK"] = str(current_conf["nod...

@<1523701435869433856:profile|SmugDolphin23> Everything worked after setting the variables: --env NCCL_IB_DISABLE=1 --env NCCL_SOCKET_IFNAME=ens192 --env NCCL_P2P_DISABLE=1. But previously, these variables were not required for a successful launch. When I run ddp training with two nodes , everything works for me now. But as soon as I increase their number ( nodes > 2 ), I get the following error.

Traceback (most recent call last):
  File "/root/.clearml/venvs-builds/3.11/code/light...

@<1523701435869433856:profile|SmugDolphin23> This error occurs when a secondary task is created with launch_multi_node. And this error disappears when I add the reuse_last_task_id=False flag when initializing the task. But now I have a new problem. I can't request more than 2 nodes. The training logs freezes after several iterations of first epoch with three workers. And if i request four workers i get this error:

DEBUG Epoch 0:   8%|▊         | 200/2484 [04:43<53:55,  0.71it/s, v_num=...

Hi @<1523701087100473344:profile|SuccessfulKoala55> where can I get examples of REST API requests for creating reports?

@<1523701435869433856:profile|SmugDolphin23> It is possible to request up to 5 workers in the toy example with Feed Forward and MNIST, BUT it is not possible to request more than 2 workers on a real large model

Hi @<1523701205467926528:profile|AgitatedDove14>
I started an experiment with gpus=2 and node=2 and I have the following logs

@<1523701087100473344:profile|SuccessfulKoala55>

@<1523701435869433856:profile|SmugDolphin23> Two tasks were created when gpus=2, nodes=2, task.launch_multi_node(node). But their running status does not end, and model training does not begin.

Hi @<1523701435869433856:profile|SmugDolphin23> ! I set NODE_RANK in the environment and now

• if gpus=2, node=2, task.launch_multi_node(node) : three tasks are created, and two of which are completed, but one is failed. In this case, are created (gpus*nodes-1) of tasks, some of which crashes with an error, or they all fall with an error. the behavior is inconsistent.
• if gpus=2, node=2, task.launch_multi_node(node*gpus) : seven tasks are created.I n this case, all tasks are failed except t...

@<1523701070390366208:profile|CostlyOstrich36>

@<1523701070390366208:profile|CostlyOstrich36> Any ideas?

thank you!

Hi @<1523701435869433856:profile|SmugDolphin23> Thank you for your reply!
I use 2 machines.
I set these parameters, but unfortunately, the training has not started.

torch.distributed.DistStoreError: Timed out after 1801 seconds waiting for clients. 2/4 clients joined.

@<1523701070390366208:profile|CostlyOstrich36> If I run the pipeline with the same input parameters, all the steps will also be re-run, nothing will be taken from the cache

I had a similar behavior: the parameters for starting the pipeline are not selected in a detailes view, only in the table view

Can you help me?

@<1523701070390366208:profile|CostlyOstrich36> i have 2 clearm-serving instances with endpoints

@<1523701435869433856:profile|SmugDolphin23> yeah, I am running this inside a docker container and cuda is available

I create a pipeline via PipelineController with adding a step as a function

pipe = PipelineController(
        name=cfg.clearml.pipeline_name,
        project=cfg.clearml.project_name,
        target_project=True,
        version=cfg.clearml.version,
        add_pipeline_tags=True,
        docker=cfg.clearml.dockerfile,
        docker_args=DefaultMLPLATparam().docker_arg,
        packages=packages,
        retry_on_failure=3
    )

for parameter in cfg.clearml.params:
        pipe.add_...

The /root/clearml.conf file no longer contains anything.

@<1523701070390366208:profile|CostlyOstrich36> Above, I provided the code for this pipeline, I specify cache_executed_step=True for each pipeline step , but it doesn't work.

@<1523701070390366208:profile|CostlyOstrich36> Any ideas?

I store my data in s3 and clearml tracks this data. I want to migrate this data from one ClearML instance to another, that is, transfer it to another s3 and have a new ClearML instance track it

Hi @<1523701205467926528:profile|AgitatedDove14>
I define a pipeline through functions. I have a lot of parameters, about 40. It is inconvenient to overwrite them all from the window that is on the screen.

@<1523701435869433856:profile|SmugDolphin23> it work with gpus=1 and node=2 and there are only two tasks is created

@<1523701435869433856:profile|SmugDolphin23> gloo doesn't work for me either

but torch work with nccl and task.launch_multi_node

problems arise specifically with pytorch-lightning

@<1523701205467926528:profile|AgitatedDove14> thanks for your reply! Do I also need to change the path for docker_pip_cache ?

None

@<1523701435869433856:profile|SmugDolphin23> Each task shows that process allocates only 1 gpu out of 2 (all task have the same scalar as below)