[Bug] RuntimeError: [3] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0', but store->get('0') got error: Socket Timeout

[kkscilife]

Describe the bug

It's a probabilistic occurrence, Socket Timeout when group.allreduce([tensor], opts)
if group in _world.pg_coalesce_state.keys():
# We are in coalescing context, do not issue single operation, just append a collective representation
coll = _CollOp(all_reduce, tensor, None, op, None)
_world.pg_coalesce_state[group].append(coll)
_world.pg_coalesce_state[group].append(coll)
_world.pg_coalesce_state[group].append(coll)
_world.pg_coalesce_state[group].append(coll)
if async_op:
return _IllegalWork()
else:
return None

  work = group.allreduce([tensor], opts)

E RuntimeError: [3] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0', but store->get('0') got error: Socket Timeout
E Exception raised from doWait at ../torch/csrc/distributed/c10d/TCPStore.cpp:445 (most recent call first):
More information: https://github.com/InternLM/InternEvo/actions/runs/10012995770/job/27889229378

Environment

python3.10
torch2.1

Other information

No response