[Installation]: Running ohereForAI/c4ai-command-r-v01 with main pytorch

[laithsakka]

Your current environment

why is it important:
This is a prerequisite to the work on enabling troch.compile on vllm, we need to be able to build vllm with nightly so that we can iterate on changes and try features that are not released yet.

current error:
Failed to import from vllm._C with ImportError('/home/lsakka/vllm/vllm/_C.abi3.so: undefined symbol: cuTensorMapEncodeTiled')

any idea what this could be?
It was mentioned that vllm was struggling to upgrade one step version

Collecting environment information...
WARNING 07-11 16:20:31 _custom_ops.py:14] Failed to import from vllm._C with ImportError('/home/lsakka/vllm/vllm/_C.abi3.so: undefined symbol: cuTensorMapEncodeTiled')
PyTorch version: 2.5.0a0+git9c9744c
Is debug build: False
CUDA used to build PyTorch: 12.1
ROCM used to build PyTorch: N/A

OS: CentOS Stream 9 (x86_64)
GCC version: (GCC) 11.4.1 20231218 (Red Hat 11.4.1-3)
Clang version: 17.0.6 (CentOS 17.0.6-5.el9)
CMake version: version 3.30.0
Libc version: glibc-2.34

Python version: 3.11.9 (main, Apr 19 2024, 16:48:06) [GCC 11.2.0] (64-bit runtime)
Python platform: Linux-5.12.0-0_fbk16_zion_7661_geb00762ce6d2-x86_64-with-glibc2.34
Is CUDA available: True
CUDA runtime version: 12.1.105
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: 
GPU 0: NVIDIA PG509-210
GPU 1: NVIDIA PG509-210
GPU 2: NVIDIA PG509-210
GPU 3: NVIDIA PG509-210
GPU 4: NVIDIA PG509-210
GPU 5: NVIDIA PG509-210
GPU 6: NVIDIA PG509-210
GPU 7: NVIDIA PG509-210

Nvidia driver version: 525.105.17
cuDNN version: Probably one of the following:
/usr/lib64/libcudnn.so.8.8.0
/usr/lib64/libcudnn_adv_infer.so.8.8.0
/usr/lib64/libcudnn_adv_train.so.8.8.0
/usr/lib64/libcudnn_cnn_infer.so.8.8.0
/usr/lib64/libcudnn_cnn_train.so.8.8.0
/usr/lib64/libcudnn_ops_infer.so.8.8.0
/usr/lib64/libcudnn_ops_train.so.8.8.0
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture:                    x86_64
CPU op-mode(s):                  32-bit, 64-bit
Address sizes:                   46 bits physical, 48 bits virtual
Byte Order:                      Little Endian
CPU(s):                          192
On-line CPU(s) list:             0-191
Vendor ID:                       GenuineIntel
Model name:                      Intel(R) Xeon(R) Platinum 8339HC CPU @ 1.80GHz
CPU family:                      6
Model:                           85
Thread(s) per core:              2
Core(s) per socket:              24
Socket(s):                       4
Stepping:                        11
Frequency boost:                 enabled
CPU(s) scaling MHz:              99%
CPU max MHz:                     1801.0000
CPU min MHz:                     800.0000
BogoMIPS:                        3600.00
Flags:                           fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 invpcid_single intel_ppin ssbd mba ibrs ibpb stibp ibrs_enhanced tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb intel_pt avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local avx512_bf16 dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req pku ospke avx512_vnni md_clear flush_l1d arch_capabilities
Virtualization:                  VT-x
L1d cache:                       3 MiB (96 instances)
L1i cache:                       3 MiB (96 instances)
L2 cache:                        96 MiB (96 instances)
L3 cache:                        132 MiB (4 instances)
NUMA node(s):                    4
NUMA node0 CPU(s):               0-23,96-119
NUMA node1 CPU(s):               24-47,120-143
NUMA node2 CPU(s):               48-71,144-167
NUMA node3 CPU(s):               72-95,168-191
Vulnerability Itlb multihit:     Not affected
Vulnerability L1tf:              Not affected
Vulnerability Mds:               Not affected
Vulnerability Meltdown:          Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1:        Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:        Mitigation; Enhanced IBRS, IBPB conditional, RSB filling
Vulnerability Srbds:             Not affected
Vulnerability Tsx async abort:   Not affected

Versions of relevant libraries:
[pip3] numpy==1.26.4
[pip3] nvidia-nccl-cu12==2.20.5
[pip3] torch==2.5.0a0+git9c9744c
[pip3] torchvision==0.18.0
[pip3] torchvision==0.19.0a0+d23a6e1
[pip3] transformers==4.42.3
[pip3] triton==3.0.0
[conda] blas                      1.0                         mkl  
[conda] mkl                       2023.1.0         h213fc3f_46344  
[conda] mkl-service               2.4.0           py311h5eee18b_1  
[conda] mkl_fft                   1.3.8           py311h5eee18b_0  
[conda] mkl_random                1.2.4           py311hdb19cb5_0  
[conda] numpy                     1.26.4          py311h08b1b3b_0  
[conda] numpy-base                1.26.4          py311hf175353_0  
[conda] nvidia-nccl-cu12          2.20.5                   pypi_0    pypi
[conda] torch                     2.5.0a0+git9c9744c           dev_0    <develop>
[conda] torchvision               0.18.0                   pypi_0    pypi
[conda] transformers              4.42.3                   pypi_0    pypi
[conda] triton                    3.0.0                    pypi_0    pypi
ROCM Version: Could not collect
Neuron SDK Version: N/A
vLLM Version: 0.5.1
vLLM Build Flags:
CUDA Archs: Not Set; ROCm: Disabled; Neuron: Disabled
GPU Topology:
GPU0    GPU1    GPU2    GPU3    GPU4    GPU5    GPU6    GPU7    CPU Affinity    NUMA Affinity
GPU0     X      NV4     SYS     SYS     NV2     NV2     SYS     NV4     0-23,96-119     0
GPU1    NV4      X      SYS     SYS     NV4     NV2     NV2     SYS     0-23,96-119     0
GPU2    SYS     SYS      X      NV4     SYS     NV4     NV2     NV2     24-47,120-143   1
GPU3    SYS     SYS     NV4      X      NV2     SYS     NV4     NV2     24-47,120-143   1
GPU4    NV2     NV4     SYS     NV2      X      NV4     SYS     SYS     48-71,144-167   2
GPU5    NV2     NV2     NV4     SYS     NV4      X      SYS     SYS     48-71,144-167   2
GPU6    SYS     NV2     NV2     NV4     SYS     SYS      X      NV4     72-95,168-191   3
GPU7    NV4     SYS     NV2     NV2     SYS     SYS     NV4      X      72-95,168-191   3

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

diff file

diff --git a/requirements-build.txt b/requirements-build.txt
index 1a07a94e..76f44f0c 100644
--- a/requirements-build.txt
+++ b/requirements-build.txt
@@ -3,5 +3,4 @@ cmake>=3.21
 ninja
 packaging
 setuptools>=49.4.0
-torch==2.3.0
 wheel
diff --git a/requirements-cpu.txt b/requirements-cpu.txt
index 754070df..8145cf37 100644
--- a/requirements-cpu.txt
+++ b/requirements-cpu.txt
@@ -2,6 +2,4 @@
 -r requirements-common.txt
 
 # Dependencies for x86_64 CPUs
-torch == 2.3.1+cpu; platform_machine != "ppc64le"
-torchvision == 0.18.1+cpu; platform_machine != "ppc64le"   # required for the image processor of phi3v, this must be updated alongside torch
 triton >= 2.2.0  # FIXME(woosuk): This is a hack to avoid import error.
diff --git a/requirements-cuda.txt b/requirements-cuda.txt
index 10596ed8..776b8338 100644
--- a/requirements-cuda.txt
+++ b/requirements-cuda.txt
@@ -4,8 +4,4 @@
 # Dependencies for NVIDIA GPUs
 ray >= 2.9
 nvidia-ml-py # for pynvml package
-torch == 2.3.0
 # These must be updated alongside torch
-torchvision == 0.18.0   # Required for phi3v processor, also see https://github.com/pytorch/vision?tab=readme-ov-file#installation for corresponding version
-xformers == 0.0.26.post1  # Requires PyTorch 2.3.0
-vllm-flash-attn == 2.5.9  # Requires PyTorch 2.3.0
diff --git a/requirements-openvino.txt b/requirements-openvino.txt
index e555d525..5012b6d4 100644
--- a/requirements-openvino.txt
+++ b/requirements-openvino.txt
@@ -2,7 +2,6 @@
 -r requirements-common.txt
 
 # OpenVINO dependencies
-torch >= 2.1.2
 openvino ~= 2024.3.0.dev
 optimum-intel[openvino] >= 1.17.2

How you are installing vllm

what did I do:

1. I updated vllm requirments files as the following and remove pinned versions of xformers, flash_attn, torch, triton. (see diff file bellow)
2. build pytorch nightly, xformers, flash_attn (python setup.py develop) and install latest triton.
3. build vllm
4. load flash_attn directly see the PR above.
   when Import vllm i get the error bellow:

current error:
Failed to import from vllm._C with ImportError('/home/lsakka/vllm/vllm/_C.abi3.so: undefined symbol: cuTensorMapEncodeTiled')

[laithsakka]

the undefined symbol cuTensorMapEncodeTiled is solved by workaround
export LD_PRELOAD=/usr/lib64/libcuda.so
``
nm -C _C.abi3.so | grep "cuTensorMap"
U cuTensorMapEncodeTiled
the workaround that you suggested to
export LD_PRELOAD=/usr/lib64/libcuda.so

[youkaichao]

import logging
File "/home/lsakka/vllm/vllm/logging/init.py", line 1, in

import logging goes to vllm/logging, which is caused by incorret python path i think. it should go to python builtin logging module.

[laithsakka]

I resolved the issue , I was running python from vllm/vllm folder and python was confused
for now I can import vllm with not issue
running
python benchmark_latency.py --model CohereForAI/c4ai-command-r-v01 -tp=4 --batch-size=1
hands crossed !

[laithsakka]

i get the following messages and running vllm stucks on the benchmark

 Found nccl from library libnccl.so.2
(VllmWorkerProcess pid=3656580) ERROR 07-14 19:17:50 pynccl_wrapper.py:196] Failed to load NCCL library from libnccl.so.2 .It is expected if you are not running on NVIDIA/AMD GPUs.Otherwise, the nccl library might not exist, be corrupted or it does not support the current platform Linux-5.12.0-0_fbk16_zion_7661_geb00762ce6d2-x86_64-with-glibc2.34.If you already have the library, please set the environment variable VLLM_NCCL_SO_PATH to point to the correct nccl library path.

th1991) [[email protected] ~/vllm/benchmarks (main)]$ python benchmark_latency.py --model CohereForAI/c4ai-command-r-v01 -tp=4 --batch-size=1
WARNING 07-14 19:17:44 _custom_ops.py:14] Failed to import from vllm._C with ImportError('/home/lsakka/vllm/vllm/_C.abi3.so: undefined symbol: cuTensorMapEncodeTiled')
Namespace(model='CohereForAI/c4ai-command-r-v01', speculative_model=None, num_speculative_tokens=None, speculative_draft_tensor_parallel_size=None, tokenizer=None, quantization=None, tensor_parallel_size=4, input_len=32, output_len=128, batch_size=1, n=1, use_beam_search=False, num_iters_warmup=10, num_iters=30, trust_remote_code=False, max_model_len=None, dtype='auto', enforce_eager=False, kv_cache_dtype='auto', quantization_param_path=None, profile=False, profile_result_dir=None, device='auto', block_size=16, enable_chunked_prefill=False, enable_prefix_caching=False, use_v2_block_manager=False, ray_workers_use_nsight=False, download_dir=None, output_json=None, gpu_memory_utilization=0.9, load_format='auto', distributed_executor_backend=None, otlp_traces_endpoint=None)
INFO 07-14 19:17:45 config.py:696] Defaulting to use mp for distributed inference
INFO 07-14 19:17:45 llm_engine.py:174] Initializing an LLM engine (v0.5.1) with config: model='CohereForAI/c4ai-command-r-v01', speculative_config=None, tokenizer='CohereForAI/c4ai-command-r-v01', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, rope_scaling=None, rope_theta=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.float16, max_seq_len=8192, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=4, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, quantization_param_path=None, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='outlines'), observability_config=ObservabilityConfig(otlp_traces_endpoint=None), seed=0, served_model_name=CohereForAI/c4ai-command-r-v01, use_v2_block_manager=False, enable_prefix_caching=False)
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
INFO 07-14 19:17:46 selector.py:191] Cannot use FlashAttention-2 backend because the vllm_flash_attn package is not found. `pip install vllm-flash-attn` for better performance.
INFO 07-14 19:17:46 selector.py:53] Using XFormers backend.
(VllmWorkerProcess pid=3656571) INFO 07-14 19:17:46 selector.py:191] Cannot use FlashAttention-2 backend because the vllm_flash_attn package is not found. `pip install vllm-flash-attn` for better performance.
(VllmWorkerProcess pid=3656580) INFO 07-14 19:17:46 selector.py:191] Cannot use FlashAttention-2 backend because the vllm_flash_attn package is not found. `pip install vllm-flash-attn` for better performance.
(VllmWorkerProcess pid=3656571) INFO 07-14 19:17:46 selector.py:53] Using XFormers backend.
(VllmWorkerProcess pid=3656580) INFO 07-14 19:17:46 selector.py:53] Using XFormers backend.
(VllmWorkerProcess pid=3656573) INFO 07-14 19:17:46 selector.py:191] Cannot use FlashAttention-2 backend because the vllm_flash_attn package is not found. `pip install vllm-flash-attn` for better performance.
(VllmWorkerProcess pid=3656573) INFO 07-14 19:17:46 selector.py:53] Using XFormers backend.
/home/lsakka/xformers/xformers/ops/fmha/flash.py:211: FutureWarning: `torch.library.impl_abstract` was renamed to `torch.library.register_fake`. Please use that instead; we will remove `torch.library.impl_abstract` in a future version of PyTorch.
  @torch.library.impl_abstract("xformers_flash::flash_fwd")
/home/lsakka/xformers/xformers/ops/fmha/flash.py:344: FutureWarning: `torch.library.impl_abstract` was renamed to `torch.library.register_fake`. Please use that instead; we will remove `torch.library.impl_abstract` in a future version of PyTorch.
  @torch.library.impl_abstract("xformers_flash::flash_bwd")
/home/lsakka/xformers/xformers/ops/swiglu_op.py:127: FutureWarning: `torch.cuda.amp.custom_fwd(args...)` is deprecated. Please use `torch.amp.custom_fwd(args..., device_type='cuda')` instead.
  @torch.cuda.amp.custom_fwd
/home/lsakka/xformers/xformers/ops/swiglu_op.py:148: FutureWarning: `torch.cuda.amp.custom_bwd(args...)` is deprecated. Please use `torch.amp.custom_bwd(args..., device_type='cuda')` instead.
  @torch.cuda.amp.custom_bwd
(VllmWorkerProcess pid=3656580) /home/lsakka/xformers/xformers/ops/fmha/flash.py:211: FutureWarning: `torch.library.impl_abstract` was renamed to `torch.library.register_fake`. Please use that instead; we will remove `torch.library.impl_abstract` in a future version of PyTorch.
(VllmWorkerProcess pid=3656580)   @torch.library.impl_abstract("xformers_flash::flash_fwd")
(VllmWorkerProcess pid=3656580) /home/lsakka/xformers/xformers/ops/fmha/flash.py:344: FutureWarning: `torch.library.impl_abstract` was renamed to `torch.library.register_fake`. Please use that instead; we will remove `torch.library.impl_abstract` in a future version of PyTorch.
(VllmWorkerProcess pid=3656580)   @torch.library.impl_abstract("xformers_flash::flash_bwd")
(VllmWorkerProcess pid=3656580) /home/lsakka/xformers/xformers/ops/swiglu_op.py:127: FutureWarning: `torch.cuda.amp.custom_fwd(args...)` is deprecated. Please use `torch.amp.custom_fwd(args..., device_type='cuda')` instead.
(VllmWorkerProcess pid=3656580)   @torch.cuda.amp.custom_fwd
(VllmWorkerProcess pid=3656580) /home/lsakka/xformers/xformers/ops/swiglu_op.py:148: FutureWarning: `torch.cuda.amp.custom_bwd(args...)` is deprecated. Please use `torch.amp.custom_bwd(args..., device_type='cuda')` instead.
(VllmWorkerProcess pid=3656580)   @torch.cuda.amp.custom_bwd
(VllmWorkerProcess pid=3656580) INFO 07-14 19:17:49 multiproc_worker_utils.py:215] Worker ready; awaiting tasks
(VllmWorkerProcess pid=3656571) /home/lsakka/xformers/xformers/ops/fmha/flash.py:211: FutureWarning: `torch.library.impl_abstract` was renamed to `torch.library.register_fake`. Please use that instead; we will remove `torch.library.impl_abstract` in a future version of PyTorch.
(VllmWorkerProcess pid=3656571)   @torch.library.impl_abstract("xformers_flash::flash_fwd")
(VllmWorkerProcess pid=3656571) /home/lsakka/xformers/xformers/ops/fmha/flash.py:344: FutureWarning: `torch.library.impl_abstract` was renamed to `torch.library.register_fake`. Please use that instead; we will remove `torch.library.impl_abstract` in a future version of PyTorch.
(VllmWorkerProcess pid=3656571)   @torch.library.impl_abstract("xformers_flash::flash_bwd")
(VllmWorkerProcess pid=3656573) /home/lsakka/xformers/xformers/ops/fmha/flash.py:211: FutureWarning: `torch.library.impl_abstract` was renamed to `torch.library.register_fake`. Please use that instead; we will remove `torch.library.impl_abstract` in a future version of PyTorch.
(VllmWorkerProcess pid=3656573)   @torch.library.impl_abstract("xformers_flash::flash_fwd")
(VllmWorkerProcess pid=3656573) /home/lsakka/xformers/xformers/ops/fmha/flash.py:344: FutureWarning: `torch.library.impl_abstract` was renamed to `torch.library.register_fake`. Please use that instead; we will remove `torch.library.impl_abstract` in a future version of PyTorch.
(VllmWorkerProcess pid=3656573)   @torch.library.impl_abstract("xformers_flash::flash_bwd")
(VllmWorkerProcess pid=3656571) /home/lsakka/xformers/xformers/ops/swiglu_op.py:127: FutureWarning: `torch.cuda.amp.custom_fwd(args...)` is deprecated. Please use `torch.amp.custom_fwd(args..., device_type='cuda')` instead.
(VllmWorkerProcess pid=3656571)   @torch.cuda.amp.custom_fwd
(VllmWorkerProcess pid=3656571) /home/lsakka/xformers/xformers/ops/swiglu_op.py:148: FutureWarning: `torch.cuda.amp.custom_bwd(args...)` is deprecated. Please use `torch.amp.custom_bwd(args..., device_type='cuda')` instead.
(VllmWorkerProcess pid=3656571)   @torch.cuda.amp.custom_bwd
(VllmWorkerProcess pid=3656571) INFO 07-14 19:17:49 multiproc_worker_utils.py:215] Worker ready; awaiting tasks
(VllmWorkerProcess pid=3656573) /home/lsakka/xformers/xformers/ops/swiglu_op.py:127: FutureWarning: `torch.cuda.amp.custom_fwd(args...)` is deprecated. Please use `torch.amp.custom_fwd(args..., device_type='cuda')` instead.
(VllmWorkerProcess pid=3656573)   @torch.cuda.amp.custom_fwd
(VllmWorkerProcess pid=3656573) /home/lsakka/xformers/xformers/ops/swiglu_op.py:148: FutureWarning: `torch.cuda.amp.custom_bwd(args...)` is deprecated. Please use `torch.amp.custom_bwd(args..., device_type='cuda')` instead.
(VllmWorkerProcess pid=3656573)   @torch.cuda.amp.custom_bwd
(VllmWorkerProcess pid=3656573) INFO 07-14 19:17:49 multiproc_worker_utils.py:215] Worker ready; awaiting tasks
(VllmWorkerProcess pid=3656571) INFO 07-14 19:17:50 utils.py:741] Found nccl from library libnccl.so.2
INFO 07-14 19:17:50 utils.py:741] Found nccl from library libnccl.so.2
(VllmWorkerProcess pid=3656571) ERROR 07-14 19:17:50 pynccl_wrapper.py:196] Failed to load NCCL library from libnccl.so.2 .It is expected if you are not running on NVIDIA/AMD GPUs.Otherwise, the nccl library might not exist, be corrupted or it does not support the current platform Linux-5.12.0-0_fbk16_zion_7661_geb00762ce6d2-x86_64-with-glibc2.34.If you already have the library, please set the environment variable VLLM_NCCL_SO_PATH to point to the correct nccl library path.
(VllmWorkerProcess pid=3656573) INFO 07-14 19:17:50 utils.py:741] Found nccl from library libnccl.so.2
ERROR 07-14 19:17:50 pynccl_wrapper.py:196] Failed to load NCCL library from libnccl.so.2 .It is expected if you are not running on NVIDIA/AMD GPUs.Otherwise, the nccl library might not exist, be corrupted or it does not support the current platform Linux-5.12.0-0_fbk16_zion_7661_geb00762ce6d2-x86_64-with-glibc2.34.If you already have the library, please set the environment variable VLLM_NCCL_SO_PATH to point to the correct nccl library path.
(VllmWorkerProcess pid=3656573) ERROR 07-14 19:17:50 pynccl_wrapper.py:196] Failed to load NCCL library from libnccl.so.2 .It is expected if you are not running on NVIDIA/AMD GPUs.Otherwise, the nccl library might not exist, be corrupted or it does not support the current platform Linux-5.12.0-0_fbk16_zion_7661_geb00762ce6d2-x86_64-with-glibc2.34.If you already have the library, please set the environment variable VLLM_NCCL_SO_PATH to point to the correct nccl library path.
(VllmWorkerProcess pid=3656580) INFO 07-14 19:17:50 utils.py:741] Found nccl from library libnccl.so.2
(VllmWorkerProcess pid=3656580) ERROR 07-14 19:17:50 pynccl_wrapper.py:196] Failed to load NCCL library from libnccl.so.2 .It is expected if you are not running on NVIDIA/AMD GPUs.Otherwise, the nccl library might not exist, be corrupted or it does not support the current platform Linux-5.12.0-0_fbk16_zion_7661_geb00762ce6d2-x86_64-with-glibc2.34.If you already have the library, please set the environment variable VLLM_NCCL_SO_PATH to point to the correct nccl library path.

[youkaichao]

how does your nightly pytorch use NCCL? static link or dynamic link? which NCCL does it use?

[laithsakka]

I fixed that by setting the version of the nccl libarary

VLLM_NCCL_SO_PATH=/home/lsakka/pytorch/build/nccl/lib/libnccl.so.2  python benchmark_latency.py --model CohereForAI/c4ai-command-r-v01 -tp=4 --batch-size=1

the benchmark still gets stuck though, i have seen local posts about nccl getting stuck and having tor revert not sure if related.
the current version i use is
NCCL version 2.21.5+cuda12.1

[youkaichao]

can you try to follow the debugging guide https://docs.vllm.ai/en/latest/getting_started/debugging.html ? there is a sanity check script to help you locate the problem.

[laithsakka]

I used the VLLM_TRACE and it get stuck at
2024-07-16 13:56:37.432732 Call to wait_until_ready in /home/lsakka/vllm/vllm/distributed/device_communicators/shm_broadcast.py:284 from create_from_process_group in /home/lsakka/vllm/vllm/distributed/device_communicators/shm_broadcast.py:489

note the following warning which i am unusre if it related

llmWorkerProcess pid=1194815) WARNING 07-16 14:31:56 custom_all_reduce.py:118] Custom allreduce is disabled because it's not supported on more than two PCIe-only GPUs. To silence this warning, specify disable_custom_all_reduce=True explicitly.
(VllmWorkerProcess pid=1194818) WARNING 07-16 14:31:56 custom_all_reduce.py:118] Custom allreduce is disabled because it's not supported on more than two PCIe-only GPUs. To silence this warning, specify disable_custom_all_reduce=True explicitly.
(VllmWorkerProcess pid=1194814) WARNING 07-16 14:31:56 custom_all_reduce.py:118] Custom allreduce is disabled because it's not supported on more than two PCIe-only GPUs. To silence this warning, specify disable_custom_all_reduce=True explicitly.
WARNING 07-16 14:31:56 custom_all_reduce.py:118] Custom allreduce is disabled because it's not supported on more than two PCIe-only GPUs. To silence this warning, specify disable_custom_all_reduce=True explicitly.

[youkaichao]

the warning is irrelevant.

can you stably reproduce this hang? if so, you can remove VLLM_TRACE_FUNCTION, and set a breakpoint in /home/lsakka/vllm/vllm/distributed/device_communicators/shm_broadcast.py:284 to see which line hangs.

that part of code sets up a publish-subscribe message queue for communication.

[eric1932]

I used the VLLM_TRACE and it get stuck at 2024-07-16 13:56:37.432732 Call to wait_until_ready in /home/lsakka/vllm/vllm/distributed/device_communicators/shm_broadcast.py:284 from create_from_process_group in /home/lsakka/vllm/vllm/distributed/device_communicators/shm_broadcast.py:489

note the following warning which i am unusre if it related

llmWorkerProcess pid=1194815) WARNING 07-16 14:31:56 custom_all_reduce.py:118] Custom allreduce is disabled because it's not supported on more than two PCIe-only GPUs. To silence this warning, specify disable_custom_all_reduce=True explicitly.
(VllmWorkerProcess pid=1194818) WARNING 07-16 14:31:56 custom_all_reduce.py:118] Custom allreduce is disabled because it's not supported on more than two PCIe-only GPUs. To silence this warning, specify disable_custom_all_reduce=True explicitly.
(VllmWorkerProcess pid=1194814) WARNING 07-16 14:31:56 custom_all_reduce.py:118] Custom allreduce is disabled because it's not supported on more than two PCIe-only GPUs. To silence this warning, specify disable_custom_all_reduce=True explicitly.
WARNING 07-16 14:31:56 custom_all_reduce.py:118] Custom allreduce is disabled because it's not supported on more than two PCIe-only GPUs. To silence this warning, specify disable_custom_all_reduce=True explicitly.

+1 hanged there at wait_until_ready()
vllm 0.5.2 run in Docker on host network

[eric1932]

I used the VLLM_TRACE and it get stuck at 2024-07-16 13:56:37.432732 Call to wait_until_ready in /home/lsakka/vllm/vllm/distributed/device_communicators/shm_broadcast.py:284 from create_from_process_group in /home/lsakka/vllm/vllm/distributed/device_communicators/shm_broadcast.py:489
note the following warning which i am unusre if it related

llmWorkerProcess pid=1194815) WARNING 07-16 14:31:56 custom_all_reduce.py:118] Custom allreduce is disabled because it's not supported on more than two PCIe-only GPUs. To silence this warning, specify disable_custom_all_reduce=True explicitly.
(VllmWorkerProcess pid=1194818) WARNING 07-16 14:31:56 custom_all_reduce.py:118] Custom allreduce is disabled because it's not supported on more than two PCIe-only GPUs. To silence this warning, specify disable_custom_all_reduce=True explicitly.
(VllmWorkerProcess pid=1194814) WARNING 07-16 14:31:56 custom_all_reduce.py:118] Custom allreduce is disabled because it's not supported on more than two PCIe-only GPUs. To silence this warning, specify disable_custom_all_reduce=True explicitly.
WARNING 07-16 14:31:56 custom_all_reduce.py:118] Custom allreduce is disabled because it's not supported on more than two PCIe-only GPUs. To silence this warning, specify disable_custom_all_reduce=True explicitly.

+1 hanged there at wait_until_ready() vllm 0.5.2 run in Docker on host network

It turns out that using host network on a IPv6 only machine is problematic. Bridged network works just fine on my side

[youkaichao]

In v0.5.2, v0.5.3, and v0.5.3.post1, there is a bug caused by zmq , which can cause hangs at a low probability (once in about 20 times, depending on the machine configuration). The solution is to upgrade to the latest version of vllm to include the fix .

[github-actions]

This issue has been automatically marked as stale because it has not had any activity within 90 days. It will be automatically closed if no further activity occurs within 30 days. Leave a comment if you feel this issue should remain open. Thank you!

[github-actions]

This issue has been automatically closed due to inactivity. Please feel free to reopen if you feel it is still relevant. Thank you!