一、测试环境

二、创建容器

1. 获取镜像

docker pull registry.iluvatar.com.cn:10443/customer/sz/vllm0.23.0-4.4.0-x86:v1

2. 创建容器

三、模型部署

#进入容器

docker exec -it hy3 bash

1. 下载权重和依赖包

通过ModelScope下载Hy3-W4A8模型权重：

pip3 install modelscope

mkdir -p /data/models

cd /data/models

modelscope download --model iluvatar-corex/Hy3-W4A8 --local_dir ./Hy3-W4A8

把下面附件拷贝到/home目录下

  vllm-0.23.0+corex.20260710012703-py3-none-any.whl
下载 详情 

2. 启动模型

#Hy3-W4A8 部署

#替代镜像里的vllm包

pip3 uninstall vllm -y

pip3 install /home/vllm-0.23.0+corex.20260710012703-py3-none-any.whl

#部署模型

export VLLM_ENFORCE_CUDA_GRAPH=1
export VLLM_ENGINE_ITERATION_TIMEOUT_S=36000
export VLLM_W8A8_MOE_USE_W4A8=1
export TORCHDYNAMO_DISABLE=1
vllm serve /data/models/Hy3-W4A8 \
 --tensor-parallel-size 4 \
 --pipeline-parallel-size 4 \
 --tool-call-parser hy_v3 \
 --enable-auto-tool-choice \
 --port 8000 \
 --served-model-name hy3

四、测试验证

# Chat
curl http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"hy3","messages":[{"role":"user","content":"介绍一下湖南省常德美食"}],"max_tokens":1024}'

# Tool call
curl http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"hy3","messages":[{"role":"user","content":"北京的天气"}],"max_tokens":128,"tools":[{"type":"function","function":{"name":"get_weather","description":"获取天气","parameters":{"type":"object","properties":{"city":{"type":"string"}},"required":["city"]}}}]}'