Ubuntu中部署Qwen 2.5 Omni多模态模型（支持视频分析）

准备工作：

a、更新Nvidia最新的驱动程序，并安装好Cuda
b、下载模型文件
(本例中通过modelscope下载：

pip install modelscope -i https://pypi.tuna.tsinghua.edu.cn/simple

export MODELSCOPE_CACHE=/home/zhangfanxp/aiproject/MyCode/

modelscope download --model Qwen/Qwen2.5-Omni-3B

)

----------------------------------------------------------------------------------------------------------

1、创建虚拟环境

conda create -n Omni3B python=3.10 -y && conda activate Omni3B

2、安装Pytorch及相关库

pip install torch==2.6.0 torchvision==0.21.0 torchaudio==2.6.0 --index-url https://download.pytorch.org/whl/cu126

3、安装cuda-toolkit

conda install -c nvidia cuda-toolkit -y

4、安装Transformers框架

pip install git+https://github.com/huggingface/transformers@3a1ead0aabed473eafe527915eea8c197d424356
（执行git+https这行命令必须可以科学上网，可以把主机接入软路由试试看，多试几次！）


pip install accelerate sentencepiece soundfile -i https://pypi.tuna.tsinghua.edu.cn/simple


5、安装qwen omni utils

pip install "qwen-omni-utils[decord]" -i https://pypi.tuna.tsinghua.edu.cn/simple

6、安装flash attn

pip install -U flash-attn --no-build-isolation -i https://pypi.tuna.tsinghua.edu.cn/simple
（Building wheel的时候时间较长，要等一会儿）

7、编写测试代码app.py

--------------------------------------------------------------------------------------------------

import torch

import soundfile as sf

from transformers import Qwen2_5OmniModel, Qwen2_5OmniProcessor

from qwen_omni_utils import process_mm_info

model = Qwen2_5OmniModel.from_pretrained(

    "/home/zhangfanxp/model/Qwen2.5-Omni-3B",

    torch_dtype=torch.bfloat16,

    device_map="auto",

    attn_implementation="flash_attention_2",

    enable_audio_output=True

)

processor = Qwen2_5OmniProcessor.from_pretrained("/home/zhangfanxp/model/Qwen2.5-Omni-3B")

conversation = [

    {

        "role": "system",

        "content": "You are Qwen, a virtual human developed by the Qwen Team, Alibaba Group, capable of perceiving auditory and visual inputs, as well as generating text and speech. Please always respond in English."

    },

    {

        "role": "user",

        "content": [{"type": "image", "image": "./002.png"}]

    }

]

text = processor.apply_chat_template(conversation, add_generation_prompt=True, tokenize=False)

audios, images, videos = process_mm_info(conversation, use_audio_in_video=True)

inputs = processor(text=text, audios=audios, images=images, videos=videos,

                   return_tensors="pt", padding=True).to(model.device).to(model.dtype)

with torch.no_grad():

    text_ids, audio = model.generate(**inputs, use_audio_in_video=True, spk="Chelsie")

output_text = processor.batch_decode(text_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0]

print("\n🌟 Qwen Omni Response:\n", output_text)

sf.write("output.wav", audio.reshape(-1).detach().cpu().numpy(), samplerate=24000)

print("\n✅ Audio saved to output.wav")

-------------------------------------------------------------------------------------------------------

执行测试代码：

python app.py

显存有限可以如代码中的示例『测试图片分析』能力，
如果，显存大于24G，可以将image改为“video”，002.png改为“001.mp4”,并且，视频文件中要包含音轨信息）

    {

        "role": "user",

        "content": [{"type": "image", "image": "./002.png"}]

    }