@<1523701435869433856:profile|SmugDolphin23> Everything worked after setting the variables: --env NCCL_IB_DISABLE=1 --env NCCL_SOCKET_IFNAME=ens192 --env NCCL_P2P_DISABLE=1. But previously, these variables were not required for a successful launch. When I run ddp training with two nodes , everything works for me now. But as soon as I increase their number ( nodes > 2 ), I get the following error.

Traceback (most recent call last):
  File "/root/.clearml/venvs-builds/3.11/code/lightning_ddp_rc.py", line 104, in <module>
    config = task.launch_multi_node(nodes, devices=gpus, hide_children=True, wait=True)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.clearml/venvs-builds/3.11/lib/python3.11/site-packages/clearml/task.py", line 2195, in launch_multi_node
    Task.enqueue(node, queue_id=self.data.execution.queue)
  File "/root/.clearml/venvs-builds/3.11/lib/python3.11/site-packages/clearml/task.py", line 1557, in enqueue
    raise exception
  File "/root/.clearml/venvs-builds/3.11/lib/python3.11/site-packages/clearml/task.py", line 1548, in enqueue
    res = cls._send(session=session, req=req)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.clearml/venvs-builds/3.11/lib/python3.11/site-packages/clearml/backend_interface/base.py", line 107, in _send
    raise SendError(res, error_msg)
clearml.backend_interface.session.SendError: Action failed <400/706: tasks.enqueue/v1.0 (Failed adding task to queue since task is already queued: task=88808574c7c648ac97bd18303c230710)> (queue=1f0eee180f3d43ddbb432badf328e85b, task=88808574c7c648ac97bd18303c230710, verify_watched_queue=False)
2024-12-04 10:42:25
Process failed, exit code 1