下载4.4.0 docker base镜像包和相关包

通过SFTP下载，SFTP下载地址和密码，请联系天数售后工程师获取

get /SDK_4.4.0/x86/sdk/corex-docker-installer-4.4.0-10.2-ubuntu20.04-llm-py3.10-x86_64.run /home

下载模型

pip install modelscope

mkdir -p /data/model/Qwen2.5-VL-32B-Instruct

modelscope download --model Qwen/Qwen2.5-VL-32B-Instruct --local_dir /data/model/Qwen2.5-VL-32B-Instruct

导入base镜像

机器上有其他驱动，需要卸载，卸载步骤参考https://ixkb.iluvatar.com.cn:9443/webdoc/view/Pub8a20978a940105aa01954611b5b818aa.html

bash corex-docker-installer-4.4.0-10.2-ubuntu20.04-llm-py3.10-x86_64.run

全选安装

创建容器

docker run -dit -v /usr/src:/usr/src -v /lib/modules:/lib/modules -v /dev:/dev -v /home:/home --network=host --name= Qwen2.5 --pid=host --ipc=host --privileged --cap-add=ALL corex:4.4.0 /bin/bash

进入容器

docker exec -it Qwen2.5 bash

开启服务

export ENABLE_FLASH_ATTENTION_WITH_HEAD_DIM_PADDING=1

python3 -m vllm.entrypoints.openai.api_server \
--model /data/model/Qwen2.5-VL-32B-Instruct \
--limit-mm-per-prompt '{"image": 5}' \
--tensor-parallel-size 4 \
--gpu-memory-utilization 0.90 \
--host 0.0.0.0 \
--port 9997 \
--dtype bfloat16 \
--max-model-len 16384 \
--trust-remote-code

测试

client 端（宿主机执行）

curl http://localhost:9997/v1/chat/completions -H "Content-Type: application/json" -d '{

"model": "/data/model/Qwen2.5-VL-32B-Instruct",

"messages": [

{

"role": "user",

"content": [

{"type": "text", "text": "描述这张图片"},

{"type": "image_url", "image_url": {"url": "https://img-s-msn-com.akamaized.net/tenant/amp/entityid/AAOEcgc.img"}}

]

}

]

如果不能连外网

准备好一张图片，例如/root/AAOEcgc.img

b64=$(base64 -w 0 /root/AAOEcgc.img)

cat > payload.json <<EOF
{
"model": "/data/model/Qwen2.5-VL-32B-Instruct",
"messages": [
{
"role": "user",
"content": [
{"type": "text", "text": "描述这张图片"},
{"type": "image_url", "image_url": {"url": "data:image/png;base64,$b64"}}
]
}
]
}
EOF

curl http://localhost:9997/v1/chat/completions -H "Content-Type: application/json" -d @payload.json