Ubuntu服务器部署VLLM推理框架后,启动VLLM服务,并让局域网通过浏览器webui界面访问
1、服务器启动服务: python -m vllm.entrypoints.openai.api_server \ --model /home/user/env/vllm/QWQ \ --gpu-memory-utilization 0.85 \ --max-num-seqs 8 \ --tensor-parallel-size 1 \ --max-model-len 4096 \ --dtype float16 \ --host 0.0.0.0 \ --port 5000 2、服务器端编写webui的python代码(webui.py): import gradio as gr import requests VLLM_API_URL = "http://192.168.1.230:5000/v1/chat/completions" MODEL_ID = "/home/user/env/vllm/QWQ" def query_vllm(user_input): headers = {"Content-Type": "application/json"} data = { "model": MODEL_ID, "messages": [{"role": "user", "content": user_input}], "temperature": 0.7 } try: # 超时设为120秒 response = requests.post(VLLM_API_URL, headers=headers, json=data, timeout=120) response.raise_for_status() ...