确保已安装docker

下载Qwen3模型权重

apt-get install git-lfs 或者yum install git-lfs 

mkdir -p /home/Qwen/

cd /home/Qwen/

git clone https://www.modelscope.cn/Qwen/Qwen3-4B.git 

git clone https://www.modelscope.cn/Qwen/Qwen3-8B.git 
git clone https://www.modelscope.cn/Qwen/Qwen3-14B.git 
git clone https://www.modelscope.cn/Qwen/Qwen3-32B.git 
git clone https://www.modelscope.cn/Qwen/Qwen3-30B-A3B.git 

git clone https://www.modelscope.cn/Qwen/Qwen3-235B-A22B.git

准备镜像

联网拉取镜像

配置全向箔镜像地址

• 拉取淄博镜像

docker pull harbor.iluvatar.com.cn:10443/saas/mr-bi150-4.3.6.llm-aarch64-ubuntu20.04-py3.10-poc-llm-infer:v1.2.3

离线导入镜像

在windows上通过FileZilla工具下载，文件->站点管理器->新站点

协议：选择SFTP-SSH File Transfer Protocol

通过SFTP下载，SFTP下载地址和密码，请联系天数售后工程师获取

在远程站点内输入/client_tmp/support/，把mr-bi150-4.3.6.llm-aarch64-ubuntu20.04-py3.10-poc-llm-infer-v1.2.3.tar文件拷贝到windows本地目录上，然后放到服务器/home目录下

执行 docker load -i /home/mr-bi150-4.3.6.llm-aarch64-ubuntu20.04-py3.10-poc-llm-infer-v1.2.3.tar

启动容器

docker run -dit -v /usr/src:/usr/src -v /lib/modules:/lib/modules -v /dev:/dev -v /home:/home --network=host --name=Qwen3 --ipc=host --privileged --cap-add=ALL --pid=host harbor.iluvatar.com.cn:10443/saas/mr-bi150-4.3.6.llm-aarch64-ubuntu20.04-py3.10-poc-llm-infer:v1.2.3 /bin/bash

入容器

docker exec -it Qwen3 bash

运行offline_inference测试

export VLLM_ENFORCE_CUDA_GRAPH=1
export VLLM_FORCE_USE_CUDA_GRAPH=1 

cd /root/apps/llm-modelzoo/inference/Qwen/vllm

python3 offline_inference.py --model /home/Qwen/Qwen3-4B/  --max-tokens 256 --temperature 0.0 --max-model-len 3096 
python3 offline_inference.py --model /home/Qwen/Qwen3-8B/  --max-tokens 256 --temperature 0.0 --max-model-len 3096 
python3 offline_inference.py --model /home/Qwen/Qwen3-14B/  --max-tokens 256 -tp 2 --temperature 0.0 --max-model-len 3096 
python3 offline_inference.py --model /home/Qwen/Qwen3-32B/  --max-tokens 256 -tp 4 --temperature 0.0 --max-model-len 3096 
python3 offline_inference.py --model /home/Qwen/Qwen3-30B-A3B/  --max-tokens 256 -tp 4 --temperature 0.0 --max-model-len 3096 

export VLLM_ENFORCE_CUDA_GRAPH=0
export VLLM_USE_V1=1

python3 offline_inference.py --model /home/Qwen/Qwen3-235B-A22B/  --max-tokens 256 -tp 16 --temperature 0.0 --max-model-len 3096 --gpu-memory-utilization 0.97

api测试

(注：当前 --max-model-len≥8192时，需要加上--max-seq-len-to-capture=8192(这个值需要和max-model-len参数值对应，例如--max-model-len 10240 --max-seq-len-to-capture=10240）

# server 端

export VLLM_ENFORCE_CUDA_GRAPH=1
export VLLM_FORCE_USE_CUDA_GRAPH=1 

python3 -m vllm.entrypoints.openai.api_server --model /home/Qwen/Qwen3-4B/ --gpu-memory-utilization 0.9 --max-num-batched-tokens 5120 --max-model-len 2048 --max-num-seqs 256 --host 0.0.0.0 --port 1234 --trust-remote-code
python3 -m vllm.entrypoints.openai.api_server --model /home/Qwen/Qwen3-8B/ --gpu-memory-utilization 0.9 --max-num-batched-tokens 5120 --max-model-len 2048 --max-num-seqs 256 --host 0.0.0.0 --port 1234 --trust-remote-code
python3 -m vllm.entrypoints.openai.api_server --model /home/Qwen/Qwen3-14B/ --gpu-memory-utilization 0.9 --max-num-batched-tokens 5120 --max-model-len 2048 --max-num-seqs 256 -tp 2 --host 0.0.0.0 --port 1234 --trust-remote-code
python3 -m vllm.entrypoints.openai.api_server --model /home/Qwen/Qwen3-32B/ --gpu-memory-utilization 0.9 --max-num-batched-tokens 5120 --max-model-len 2048 --max-num-seqs 256 -tp 4 --host 0.0.0.0 --port 1234 --trust-remote-code

不加这两个参数export VLLM_ENFORCE_CUDA_GRAPH=1
export VLLM_FORCE_USE_CUDA_GRAPH=1 

运行下面这两个模型
python3 -m vllm.entrypoints.openai.api_server --model /home/Qwen/Qwen3-30B-A3B/ --gpu-memory-utilization 0.9 --max-num-batched-tokens 8192 --max-model-len 8192 --max-seq-len-to-capture=8192 --max-num-seqs 256 -tp 4 --host 0.0.0.0 --port 1234 --trust-remote-code

export VLLM_ENFORCE_CUDA_GRAPH=0

python3 -m vllm.entrypoints.openai.api_server --model /home/Qwen/Qwen3-235B-A22B --gpu-memory-utilization 0.99 --max-model-len 4096 --tensor-parallel-size 4 --pipeline-parallel-size 4  --host 0.0.0.0 --port 12345 --trust-remote-code --dtype float16

# client 端

curl -X POST http://0.0.0.0:1234/v1/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer YOUR_API_KEY" \
-d '{"model": "/home/Qwen/Qwen3-4B/","prompt": "介绍一下湖南省常德市","temperature": 0.0,"max_tokens": 512}'

curl -X POST http://0.0.0.0:1234/v1/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer YOUR_API_KEY" \
-d '{"model": "/home/Qwen/Qwen3-8B/","prompt": "介绍一下湖南省常德市","temperature": 0.0,"max_tokens": 512}'

curl -X POST http://0.0.0.0:1234/v1/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer YOUR_API_KEY" \
-d '{"model": "/home/Qwen/Qwen3-14B/","prompt": "介绍一下湖南省常德市","temperature": 0.0,"max_tokens": 512}'

curl -X POST http://0.0.0.0:1234/v1/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer YOUR_API_KEY" \
-d '{"model": "/home/Qwen/Qwen3-32B/","prompt": "介绍一下湖南省常德市","temperature": 0.0,"max_tokens": 512}'

curl -X POST http://0.0.0.0:1234/v1/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer YOUR_API_KEY" \
-d '{"model": "/home/Qwen/Qwen3-30B-A3B/","prompt": "介绍一下湖南省常德市","temperature": 0.0,"max_tokens": 512}'

curl -X POST http://0.0.0.0:1234/v1/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer YOUR_API_KEY" \
-d '{"model": "/home/Qwen/Qwen3-235B-A22B/","prompt": "介绍一下湖南省常德市","temperature": 0.0,"max_tokens": 512}'

Postman测试

1、选择POST，地址栏输入http://10.209.0.31:1234/v1/completions（根据host主机实际IP修改）；

2、选择Body、raw，body输入{"model": "/home/Qwen/Qwen3-4B/","prompt": "介绍一下湖南省常德市","temperature": 0.0,"max_tokens": 512}测试；