vLLM
Open-source high-throughput inference and serving engine for large language models.
HN commenters use it for high-throughput batched serving1234 and for production inference, citing broad ecosystem support.567 The most common complaints are a hard, tweak-heavy setup891011 and frustrating bugs.121314
Why people recommend it
Batched throughput · 3
Commenters pick it for concurrent, batched serving and for getting more performance out of hardware.
“Need batching + highest total throughoutput? vLLM, complicated to deploy and install though”
“I thought the main advantage of VLLM was better concurrency management (better batching).”
Show 1 more
“We built a custom inference layer on top of vLLM”
Production use · 3
Teams run it for production inference and cite its ecosystem support.
“For LLM inference in production we use vLLM (with our own model).”
“We use vllm as it generally has the best ecosystem support.”
Show 1 more
“vLLM gang all the way!”
Cloud deployment · 3
Commenters run it on rented cloud GPUs or through a Kubernetes operator.
“I spin up a gpu instance in a cloud, run my model via vllm, connect to it via an ssh tunnel. done.”
“you need to run vllm and a model”
Show 1 more
“using just the ollama, llama.cpp containers and especially vllm operator”
Prompt caching · 2
Some users switched from llama.cpp to avoid its prompt cache bugs with Qwen and Gemma models.
“I switched from llama.cpp to vLLM because of prompt cache bugs in qwen/gemma models”
“OpenCode + vllm, model will depend on your hardware”
Quantization options · 2
Commenters point to quantization support such as turboquant and third-party libraries like glq.
“vLLM apparently already has an implementation of turboquant available”
“Glq runs with vLLM.”
Caveats and dissent
Setup effort · 3
Installation and per-model configuration are described as complicated or tedious.
“Need batching + highest total throughoutput? vLLM, complicated to deploy and install though”
“vLLM is very painful, even with the Spark community docker image.”
Show 1 more
“I wanted to experiment with running something a little more "low level".”
Bugs · 3
Users report broken releases, garbage output, hangs, and messy reasoning_content handling.
“I love vLLM, but damn if it isn't frustratingly buggy.”
“vLLM seems to have a lot more gotchas and kludged together stuff”
Show 1 more
“I was hoping to see the reasoning_content mess get robustly fixed”
Startup and latency · 2
Startup is slow, and one team measured slower generation for single-to-few-user workloads.
“vLLM is very painful, even with the Spark community docker image.”
“We did run vLLM on the 3090s — measured ~3 tok/s slower on generation for our single-to-few-user pattern”
Quant formats · 3
GGUF and 3-bit dynamic quant support are weak, and sharing weights with llama.cpp needs conversion.
“I only wish gguf format had better vllm support.”
“vLLM is very painful, even with the Spark community docker image.”
Show 1 more
“I mostly end up using vLLM these days”
Dependency builds · 1
One user could not build the container with updated dependencies and switched to llama.cpp.
“You still can't build the vllm container with updated dependencies since llmlite got pwned.”
GPU memory · 1
One RTX 5090 owner ran out of GPU memory.
“I already tried using vLLM but it ran out of GPU memory.”
Often compared with
1 moved from llama.cpp to vLLM
2 moved from vLLM to llama.cpp
1 moved from Ollama to vLLMSGLang3 comments mention both
1 moved from vLLM to SGLang
1 moved from vLLM to llama-server
Who recommends it in their own words
From the makers not counted in the ranking
“I worked on the vLLM feature”
“now working on block-FP8 GEMM support for Ada GPUs (L4/L40S/4090) in vLLM”
Threads where it came up
- vLLM v0.28.0 3 mentions
- Qwen 3.8 27B is out: open weights, best local dense model yet 2 mentions
- Ask HN: How do you use LLMs for private discussions? 2 mentions
- Fixing Tool-Call Accuracy in vLLM 1 mention
- Ask HN: Who wants to be hired? (October 2026) 1 mention
- Lichen – A local, BOY model system1 (Jev) server with image support 1 mention
Data: vllm.json. Quotes are verbatim; the verdict and theme labels are written by a model from these comments, and every clause links to its sources.
All comments 40
“Served from vLLM with the NVFP4 build, it scores 204”
“A dual R9700 rig is really the sweet spot right now with the vLLM-radiance fork.”
“vLLM has a PR very close to merging”
“vLLM PR nearing merge readiness for Jev like capabilities with diffusion models”
“I have a quirky vLLM on k8s on 2x OEM sparks setup with 9 models available to me.”
“the vLLM patch to turn DiffusionGemma into Jev is available”
“served by llama.cpp, Freetoken, or vLLM”
“there's llama.cpp and vLLM, each harder to setup but opens the door for squeezing more performance out of your hardware”
“I was hoping to see the reasoning_content mess get robustly fixed”
“vLLM seems to have a lot more gotchas and kludged together stuff”
“I love vLLM, but damn if it isn't frustratingly buggy.”
“vLLM gang all the way!”
“I thought the main advantage of VLLM was better concurrency management (better batching).”
“Glq runs with vLLM.”
“vllm on 4x 5090 is getting ~20 tok/s with mtp on”
“I already tried using vLLM but it ran out of GPU memory.”
“I mostly end up using vLLM these days”
“hooked it up to vLLM on Modal”
“I’ve been running a custom VLLM image with b12x”
“For LLM inference in production we use vLLM (with our own model).”
“We use vllm as it generally has the best ecosystem support.”
“w/ the big cohere-transcribe running on a local vLLM instance”
“Preliminary analysis via lm-evaluation-harness + vllm”
“you need to run vllm and a model”
“I spin up a gpu instance in a cloud, run my model via vllm, connect to it via an ssh tunnel. done.”
“vLLM is very painful, even with the Spark community docker image.”
“I have tried various self-hosted models using both vllm and llama.cpp.”
“We did run vLLM on the 3090s — measured ~3 tok/s slower on generation for our single-to-few-user pattern”
“I switch between vLLM using nvidia/Gemma-4-31B-IT-NVFP4 and llama.cpp”
“using just the ollama, llama.cpp containers and especially vllm operator”
“I switched from llama.cpp to vLLM because of prompt cache bugs in qwen/gemma models”
“OpenCode + vllm, model will depend on your hardware”
“You still can't build the vllm container with updated dependencies since llmlite got pwned.”
“vLLM apparently already has an implementation of turboquant available”
“I only wish gguf format had better vllm support.”
“The whole thing is served on a single 4090 GPU using vLLM.”
“Comparable inference speeds to vLLM”
“I wanted to experiment with running something a little more "low level".”
“We built a custom inference layer on top of vLLM”
“Need batching + highest total throughoutput? vLLM, complicated to deploy and install though”