一、测试条件
sdk4.4版、X86环境
、MR100卡、镜像corex-docker-installer-4.4.0-10.2-ubuntu20.04-llm-py3.10-x86_64.run
二、创建容器
1、获取镜像
sftp -P 29880 iluvatar_mr@iftp.iluvatar.com.cn 访问密码请咨询天数售后
get /SDK_4.4.0/x86/sdk/corex-docker-installer-4.4.0-10.2-ubuntu20.04-llm-py3.10-x86_64.run
2、镜像部署
bash corex-docker-installer-4.4.0-10.2-ubuntu20.04-llm-py3.10-x86_64.run
3、创建容器
docker run -dit -v /usr/src:/usr/src -v /lib/modules:/lib/modules -v /dev:/dev -v /data:/data -v /home:/home --network=host --name=sdk440llm --ipc=host --privileged --cap-add=ALL --pid=host 41fb7447b7b9 /bin/bash
三、启动服务
1、魔搭上下载权重
pip install modelscope
#Qwen3-Coder-Next
modelscope download --model Qwen/Qwen3-Coder-Next --local_dir Qwen3-Coder-Next
#Qwen3-Coder-Next-W4A8量化版
modelscope download --model iluvatar-corex/Qwen3-Coder-Next-W4A8 --local_dir /data/models/Qwen3-Coder-Next-W4A8
2、启动模型
#非量化版
export VLLM_ENFORCE_CUDA_GRAPH=1
export VLLM_KV_DISABLE_CROSS_GROUP_SHARE=1
vllm serve /data/models/Qwen3-Coder-Next -tp 8 --enable-auto-tool-choice --tool-call-parser qwen3_coder --compilation-config '{"cudagraph_mode": "FULL_DECODE_ONLY", "level": 0}'
#w4a8量化版
export VLLM_ENFORCE_CUDA_GRAPH=1
export VLLM_W8A8_MOE_USE_W4A8=1
export VLLM_KV_DISABLE_CROSS_GROUP_SHARE=1
vllm serve /data/models/Qwen3-Coder-Next-W4A8 -tp 2 --enable-auto-tool-choice --tool-call-parser qwen3_coder --compilation-config '{"cudagraph_mode": "FULL_DECODE_ONLY", "level": 0}'
四、curl请求测试
1、普通Curl请求测试
#非量化版
-H "Content-Type: application/json" \
-d '{
"model": "/data/models/Qwen3-Coder-Next",
"messages": [
{
"role": "user",
"content": "介绍一下人工智能"
}
],
"temperature": 0.0,
"max_tokens": 512
}'
#量化版
-H "Content-Type: application/json" \
-d '{
"model": "/data/models/Qwen3-Coder-Next-W4A8",
"messages": [
{
"role": "user",
"content": "介绍一下人工智能"
}
],
"temperature": 0.0,
"max_tokens": 512
}'
2、使用jq格式化curl输出
1)、安装jq
apt-get update
apt-get install jq
2）、curl请求测试
#非量化版
-H "Content-Type: application/json" \
-d '{
"model": "/data/models/Qwen3-Coder-Next",
"messages": [
{
"role": "user",
"content": "请介绍一下人工智能"
}
],
"temperature": 0.0,
"max_tokens": 512
}' | jq -r '.choices[0].message.content'
#量化版
-H "Content-Type: application/json" \
-d '{
"model": "/data/models/Qwen3-Coder-Next-W4A8",
"messages": [
{
"role": "user",
"content": "请介绍一下人工智能"
}
],
"temperature": 0.0,
"max_tokens": 512
}' | jq -r '.choices[0].message.content'
五、Benchmark测试
1、benchmark工具下载
vllm_benchmark.tar.gz
sftp -P 29880 iluvatar_mr@iftp.iluvatar.com.cn
访问密码请咨询天数售后
get /client_tmp/support/kk/vllm_benchmark.tar.gz /home/
2、进入容器，执行测试
docker exec -it sdk440llm bash
cd /data/models/vllm_benchmark
#非量化版
python3 benchmark_serving.py --model /data/models/Qwen3-Coder-Next --dataset-name random --random-input-len 4096 --random-output-len 4096 --num-prompts 1 --host "0.0.0.0" --port 8000
#量化版
python3 benchmark_serving.py --model /data/models/Qwen3-Coder-Next-W4A8 --dataset-name random --random-input-len 4096 --random-output-len 4096 --num-prompts 1 --host "0.0.0.0" --port 8000
3、测试结果
#非量化版
============ Serving Benchmark Result ============
Successful requests: 1
Benchmark duration (s): 198.10
Total input tokens: 4096
Total generated tokens: 4096
Request throughput (req/s): 0.01
Output token throughput (tok/s): 20.68
Total Token throughput (tok/s): 41.35
---------------Time to First Token----------------
Mean TTFT (ms): 814.09
Median TTFT (ms): 814.09
P99 TTFT (ms): 814.09
------------------Generate time-------------------
Mean LATENCY (ms): 198102.38
Median LATENCY (ms): 198102.38
P99 LATENCY (ms): 198102.38
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 48.18
Median TPOT (ms): 48.18
P99 TPOT (ms): 48.18
---------------Inter-token Latency----------------
Mean ITL (ms): 48.18
Median ITL (ms): 48.13
P99 ITL (ms): 50.15
==================================================
#w4a8量化版
============ Serving Benchmark Result ============
Successful requests: 1
Benchmark duration (s): 180.01
Total input tokens: 4096
Total generated tokens: 4096
Request throughput (req/s): 0.01
Output token throughput (tok/s): 22.75
Total Token throughput (tok/s): 45.51
---------------Time to First Token----------------
Mean TTFT (ms): 668.02
Median TTFT (ms): 668.02
P99 TTFT (ms): 668.02
------------------Generate time-------------------
Mean LATENCY (ms): 180013.66
Median LATENCY (ms): 180013.66
P99 LATENCY (ms): 180013.66
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 43.80
Median TPOT (ms): 43.80
P99 TPOT (ms): 43.80
---------------Inter-token Latency----------------
Mean ITL (ms): 43.90
Median ITL (ms): 43.83
P99 ITL (ms): 68.25
==================================================
