Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug] Unable to use Ctrl+C to normally end service on the Ascend platform #2564

Closed
3 tasks done
jiajie-yang opened this issue Oct 9, 2024 · 3 comments · Fixed by #2588
Closed
3 tasks done

[Bug] Unable to use Ctrl+C to normally end service on the Ascend platform #2564

jiajie-yang opened this issue Oct 9, 2024 · 3 comments · Fixed by #2588
Assignees
@CyCle1024 @jinminxi104

Comments

@jiajie-yang
Copy link
Contributor

Checklist

  • 1. I have searched related issues but cannot get the expected help.
  • 2. The bug has not been fixed in the latest version.
  • 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.

Describe the bug

On the Ascend platform (such as ascend910b), after the lmdeploy serve api_server command is used to start the service, the service cannot be stopped gracefully by pressing Ctrl+C. Either an error is reported or the process is stuck.

Reproduction

ASCEND_RT_VISIBLE_DEVICES=3,4 lmdeploy serve api_server \
    models/InternVL2-8B \
    --tp 2 \
    --server-port 23333 \
    --device ascend \
    --cache-max-entry-count 0.8 \
    --log-level DEBUG

Environment

[W compiler_depend.ts:623] Warning: expandable_segments currently defaults to false. You can enable this feature by `export PYTORCH_NPU_ALLOC_CONF = expandable_segments:True`. (function operator())
sys.platform: linux
Python: 3.10.14 (main, May  6 2024, 19:36:58) [GCC 11.2.0]
CUDA available: False
MUSA available: False
numpy_random_seed: 2147483648
GCC: gcc (Ubuntu 9.4.0-1ubuntu1~20.04.2) 9.4.0
PyTorch: 2.1.0
PyTorch compiling details: PyTorch built with:
  - GCC 10.2
  - C++ Version: 201703
  - Intel(R) MKL-DNN v3.1.1 (Git Hash 64f6bcbcbab628e96f33a62c3e975f8535a7bde4)
  - OpenMP 201511 (a.k.a. OpenMP 4.5)
  - LAPACK is enabled (usually provided by MKL)
  - NNPACK is enabled
  - CPU capability usage: NO AVX
  - Build settings: BLAS_INFO=open, BUILD_TYPE=Release, CXX_COMPILER=/opt/rh/devtoolset-10/root/usr/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=0 -fabi-version=11 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOCUPTI -DLIBKINETO_NOROCTRACER -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=old-style-cast -Wno-invalid-partial-specialization -Wno-unused-private-field -Wno-aligned-allocation-unavailable -Wno-missing-braces -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Werror=cast-function-type -Wno-stringop-overflow, LAPACK_INFO=open, TORCH_DISABLE_GPU_ASSERTS=ON, TORCH_VERSION=2.1.0, USE_CUDA=OFF, USE_CUDNN=OFF, USE_EIGEN_FOR_BLAS=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=OFF, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=OFF, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, 

TorchVision: 0.16.0
LMDeploy: 0.6.1+2e49fc3
transformers: 4.44.1
gradio: Not Found
fastapi: 0.115.0
pydantic: 2.9.2
triton: Not Found

Error traceback

2024-10-09 02:35:34,096 - lmdeploy - DEBUG - utils.py:220 - <SamplingLogits> take time: 2.02 ms
2024-10-09 02:35:34,097 - lmdeploy - DEBUG - utils.py:220 - <UpdateRunning> take time: 0.05 ms
^CException in thread Thread-1:
Traceback (most recent call last):
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
Process ForkServerProcess-19:6:
Process ForkServerProcess-19:5:
    self.run()
Process ForkServerProcess-19:3:
Process ForkServerProcess-19:7:
  File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/utils/multiprocess_util.py", line 91, in run
Process ForkServerProcess-19:4:
Process ForkServerProcess-19:8:
Process ForkServerProcess-19:2:
    key, func, args, kwargs = self.task_q.get(timeout=TIMEOUT)
  File "<string>", line 2, in get
Process ForkServerProcess-19:9:
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/managers.py", line 818, in _callmethod
    kind, result = conn.recv()
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/connection.py", line 250, in recv
    buf = self._recv_bytes()
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/connection.py", line 414, in _recv_bytes
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
    buf = self._recv(4)
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/connection.py", line 383, in _recv
  File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 62, in wrapper
    func(*args, **kwargs)
  File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 62, in wrapper
    func(*args, **kwargs)
  File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 62, in wrapper
    func(*args, **kwargs)
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 262, in task_distribute
    key, func_name, detail = resource_proxy[TASK_QUEUE].get()
  File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 262, in task_distribute
    key, func_name, detail = resource_proxy[TASK_QUEUE].get()
  File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 262, in task_distribute
    key, func_name, detail = resource_proxy[TASK_QUEUE].get()
  File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 62, in wrapper
    func(*args, **kwargs)
  File "<string>", line 2, in get
  File "<string>", line 2, in get
  File "<string>", line 2, in get
  File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 262, in task_distribute
    key, func_name, detail = resource_proxy[TASK_QUEUE].get()
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/managers.py", line 818, in _callmethod
    kind, result = conn.recv()
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/managers.py", line 818, in _callmethod
    kind, result = conn.recv()
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/managers.py", line 818, in _callmethod
    kind, result = conn.recv()
  File "<string>", line 2, in get
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/connection.py", line 250, in recv
    buf = self._recv_bytes()
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/connection.py", line 250, in recv
    buf = self._recv_bytes()
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/connection.py", line 250, in recv
    buf = self._recv_bytes()
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/managers.py", line 818, in _callmethod
    kind, result = conn.recv()
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/connection.py", line 414, in _recv_bytes
    buf = self._recv(4)
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/connection.py", line 414, in _recv_bytes
    buf = self._recv(4)
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/connection.py", line 414, in _recv_bytes
    buf = self._recv(4)
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/connection.py", line 250, in recv
    buf = self._recv_bytes()
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/connection.py", line 379, in _recv
    chunk = read(handle, remaining)
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/connection.py", line 379, in _recv
    chunk = read(handle, remaining)
Traceback (most recent call last):
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/connection.py", line 379, in _recv
    chunk = read(handle, remaining)
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/connection.py", line 414, in _recv_bytes
    buf = self._recv(4)
KeyboardInterrupt
KeyboardInterrupt
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/connection.py", line 379, in _recv
    chunk = read(handle, remaining)
KeyboardInterrupt
KeyboardInterrupt
    raise EOFError
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
Traceback (most recent call last):
Traceback (most recent call last):
EOFError
  File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 62, in wrapper
    func(*args, **kwargs)
  File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 262, in task_distribute
    key, func_name, detail = resource_proxy[TASK_QUEUE].get()
  File "<string>", line 2, in get
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/managers.py", line 818, in _callmethod
    kind, result = conn.recv()
Traceback (most recent call last):
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/connection.py", line 250, in recv
    buf = self._recv_bytes()
  File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 62, in wrapper
    func(*args, **kwargs)
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 62, in wrapper
    func(*args, **kwargs)
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/connection.py", line 414, in _recv_bytes
    buf = self._recv(4)
  File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 262, in task_distribute
    key, func_name, detail = resource_proxy[TASK_QUEUE].get()
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 262, in task_distribute
    key, func_name, detail = resource_proxy[TASK_QUEUE].get()
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/connection.py", line 379, in _recv
    chunk = read(handle, remaining)
  File "<string>", line 2, in get
  File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 62, in wrapper
    func(*args, **kwargs)
  File "<string>", line 2, in get
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/managers.py", line 818, in _callmethod
    kind, result = conn.recv()
KeyboardInterrupt
  File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 262, in task_distribute
    key, func_name, detail = resource_proxy[TASK_QUEUE].get()
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/managers.py", line 818, in _callmethod
    kind, result = conn.recv()
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/connection.py", line 250, in recv
    buf = self._recv_bytes()
  File "<string>", line 2, in get
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/connection.py", line 250, in recv
    buf = self._recv_bytes()
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/connection.py", line 414, in _recv_bytes
    buf = self._recv(4)
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/managers.py", line 818, in _callmethod
    kind, result = conn.recv()
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/connection.py", line 414, in _recv_bytes
    buf = self._recv(4)
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/connection.py", line 379, in _recv
    chunk = read(handle, remaining)
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/connection.py", line 250, in recv
    buf = self._recv_bytes()
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/connection.py", line 379, in _recv
    chunk = read(handle, remaining)
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/connection.py", line 414, in _recv_bytes
    buf = self._recv(4)
KeyboardInterrupt
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/connection.py", line 379, in _recv
    chunk = read(handle, remaining)
KeyboardInterrupt
KeyboardInterrupt
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
Process ForkServerProcess-3:
Process ForkServerProcess-8:
Process ForkServerProcess-4:
Process ForkServerProcess-7:
Process ForkServerProcess-9:
Process ForkServerProcess-5:
Process ForkServerProcess-6:
Process ForkServerProcess-2:
    self.run()
  File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/utils/multiprocess_util.py", line 91, in run
    key, func, args, kwargs = self.task_q.get(timeout=TIMEOUT)
  File "<string>", line 2, in get
Traceback (most recent call last):
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/managers.py", line 818, in _callmethod
Traceback (most recent call last):
Traceback (most recent call last):
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
Traceback (most recent call last):
  File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 62, in wrapper
    func(*args, **kwargs)
  File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 62, in wrapper
    func(*args, **kwargs)
  File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 62, in wrapper
    func(*args, **kwargs)
    kind, result = conn.recv()
Traceback (most recent call last):
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
  File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 262, in task_distribute
    key, func_name, detail = resource_proxy[TASK_QUEUE].get()
  File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 262, in task_distribute
    key, func_name, detail = resource_proxy[TASK_QUEUE].get()
  File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 262, in task_distribute
    key, func_name, detail = resource_proxy[TASK_QUEUE].get()
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/connection.py", line 250, in recv
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "<string>", line 2, in get
  File "<string>", line 2, in get
  File "<string>", line 2, in get
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 62, in wrapper
    func(*args, **kwargs)
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
    buf = self._recv_bytes()
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/managers.py", line 818, in _callmethod
    kind, result = conn.recv()
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/managers.py", line 818, in _callmethod
    kind, result = conn.recv()
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/managers.py", line 818, in _callmethod
    kind, result = conn.recv()
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 262, in task_distribute
    key, func_name, detail = resource_proxy[TASK_QUEUE].get()
  File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 62, in wrapper
    func(*args, **kwargs)
  File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 62, in wrapper
    func(*args, **kwargs)
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/connection.py", line 414, in _recv_bytes
  File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 62, in wrapper
    func(*args, **kwargs)
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/connection.py", line 250, in recv
    buf = self._recv_bytes()
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/connection.py", line 250, in recv
    buf = self._recv_bytes()
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/connection.py", line 250, in recv
    buf = self._recv_bytes()
  File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 62, in wrapper
    func(*args, **kwargs)
  File "<string>", line 2, in get
  File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 262, in task_distribute
    key, func_name, detail = resource_proxy[TASK_QUEUE].get()
  File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 262, in task_distribute
    key, func_name, detail = resource_proxy[TASK_QUEUE].get()
  File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 262, in task_distribute
    key, func_name, detail = resource_proxy[TASK_QUEUE].get()
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/connection.py", line 414, in _recv_bytes
    buf = self._recv(4)
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/connection.py", line 414, in _recv_bytes
    buf = self._recv(4)
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/connection.py", line 414, in _recv_bytes
    buf = self._recv(4)
    buf = self._recv(4)
  File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 262, in task_distribute
    key, func_name, detail = resource_proxy[TASK_QUEUE].get()
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/managers.py", line 818, in _callmethod
    kind, result = conn.recv()
  File "<string>", line 2, in get
  File "<string>", line 2, in get
  File "<string>", line 2, in get
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/connection.py", line 379, in _recv
    chunk = read(handle, remaining)
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/connection.py", line 379, in _recv
    chunk = read(handle, remaining)
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/connection.py", line 379, in _recv
    chunk = read(handle, remaining)
  File "<string>", line 2, in get
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/connection.py", line 383, in _recv
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/connection.py", line 250, in recv
    buf = self._recv_bytes()
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/managers.py", line 818, in _callmethod
    kind, result = conn.recv()
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/managers.py", line 818, in _callmethod
    kind, result = conn.recv()
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/managers.py", line 818, in _callmethod
    kind, result = conn.recv()
KeyboardInterrupt
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/managers.py", line 818, in _callmethod
    kind, result = conn.recv()
KeyboardInterrupt
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/connection.py", line 414, in _recv_bytes
    buf = self._recv(4)
KeyboardInterrupt
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/connection.py", line 250, in recv
    buf = self._recv_bytes()
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/connection.py", line 250, in recv
    buf = self._recv_bytes()
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/connection.py", line 250, in recv
    buf = self._recv_bytes()
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/connection.py", line 250, in recv
    buf = self._recv_bytes()
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/connection.py", line 379, in _recv
    chunk = read(handle, remaining)
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/connection.py", line 414, in _recv_bytes
    buf = self._recv(4)
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/connection.py", line 414, in _recv_bytes
    buf = self._recv(4)
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/connection.py", line 414, in _recv_bytes
    buf = self._recv(4)
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/connection.py", line 414, in _recv_bytes
    buf = self._recv(4)
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/connection.py", line 379, in _recv
    chunk = read(handle, remaining)
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/connection.py", line 379, in _recv
    chunk = read(handle, remaining)
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/connection.py", line 379, in _recv
    chunk = read(handle, remaining)
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/connection.py", line 379, in _recv
    chunk = read(handle, remaining)
KeyboardInterrupt
KeyboardInterrupt
KeyboardInterrupt
KeyboardInterrupt
KeyboardInterrupt
    raise EOFError
EOFError
INFO:     Shutting down
INFO:     Waiting for application shutdown.
INFO:     Application shutdown complete.
INFO:     Finished server process [1381088]
Traceback (most recent call last):
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/managers.py", line 810, in _callmethod
    conn = self._tls.connection
AttributeError: 'ForkAwareLocal' object has no attribute 'connection'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/util.py", line 300, in _run_finalizers
    finalizer()
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/util.py", line 224, in __call__
    res = self._callback(*self._args, **self._kwargs)
  File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 53, in wrapper
    return func(cls, *args, **kwargs)
  File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 202, in finalize
    cls.cann_kb_mgr.finalize()
  File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/cann_kb_manager/knowledge_bank_manager.py", line 42, in finalize
    self.__static_mgr.close()
  File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/cann_kb_manager/base_manager.py", line 333, in close
    for name in copy.deepcopy(self.write_kb.keys()):
  File "<string>", line 2, in keys
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/managers.py", line 814, in _callmethod
    self._connect()
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/managers.py", line 801, in _connect
    conn = self._Client(self._token.address, authkey=self._authkey)
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/connection.py", line 502, in Client
    c = SocketClient(address)
  File "/root/miniconda3/envs/lmdeploy-for-ascend/lib/python3.10/multiprocessing/connection.py", line 630, in SocketClient
    s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
^C
^C^C^C^C
@jiajie-yang
Copy link
Contributor Author

I have to manually send a SIGINT signal to the lmdeploy process to gracefully end the service.

@CyCle1024
Copy link
Collaborator

@jiajie-yang Can you comment this line:

lmdeploy/lmdeploy/pytorch/engine/model_agent.py

Line 630 in 231e5bb

atexit.register(_exit_by_sending_exit_flag, rank, self)

And then try it again?

@jiajie-yang
Copy link
Contributor Author

@jiajie-yang Can you comment this line:

lmdeploy/lmdeploy/pytorch/engine/model_agent.py

Line 630 in 231e5bb

atexit.register(_exit_by_sending_exit_flag, rank, self)

And then try it again?

I tried it. It worked! Thanks a lot!

@CyCle1024 CyCle1024 linked a pull request Oct 11, 2024 that will close this issue
fix: make exit_flag verification for ascend more general #2588
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@CyCle1024 CyCle1024

@jinminxi104 jinminxi104

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix: make exit_flag verification for ascend more general DeepLink-org/lmdeploy
3 participants
@CyCle1024 @jinminxi104 @jiajie-yang