{"slug": "vllm", "name": "vLLM", "category": "ai", "category_name": "AI models and assistants", "description": "Open-source high-throughput inference and serving engine for large language models.", "recommenders": 10, "commenters": 32, "warners": 6, "score": 5.798, "rising": 0.71, "first": "2025-11-01", "latest": "2026-09-30", "why": "HN commenters use it for high-throughput batched serving and for production inference, citing broad ecosystem support.", "why_is_quote": false, "has_verdict": true, "sentiment": {"recommend": 11, "neutral": 15, "compare": 4, "warn": 6}, "spark": [1, 0, 0, 0, 0, 1, 1, 2, 3, 1, 2, 0], "listed": true, "loved": ["Batched throughput", "Production use"], "caveat": "Setup effort", "short_quote": "there's llama.cpp and vLLM, each harder to setup but opens the door for squeezing more performance out of your hardware", "icon": true, "domain": null, "github_repo": "github.com/vllm-project/vllm", "official_url": "https://vllm.ai", "pricing": "open source", "aliases": ["vLLM"], "verdict": [{"text": "HN commenters use it for high-throughput batched serving", "cites": [{"id": 732496465, "item_id": 45781029, "url": "https://news.ycombinator.com/item?id=45781029", "by": "embedding-shape", "date": "2025-11-01", "stance": "recommend", "quote": "Need batching + highest total throughoutput? vLLM, complicated to deploy and install though", "excerpt": "Not parent, but frequent user of GPT-OSS, tried all different ways of running it. Choice goes something like this: - Need batching + highest total throughoutput? vLLM, complicated to deploy and install though, need special versions for top performance with GPT-OSS - Easiest to manage + fast enough: llama.cpp, easier to deploy as well (just a binary) and super fast, getting ~260 tok/s on a RTX Pro 6000 for the 20B ver", "context": "", "use_case": "high-throughput batched serving", "switched_from": "", "story": "Ask HN: Who uses open LLMs and coding assistants locally? Share setup and laptop"}, {"id": 790464065, "item_id": 49404004, "url": "https://news.ycombinator.com/item?id=49404004", "by": "JacobJack", "date": "2026-08-22", "stance": "compare", "quote": "I thought the main advantage of VLLM was better concurrency management (better batching).", "excerpt": "> And the comparisons in this post are not going to be running some 2.58-bit-gguf-in-ollama with a couple test prompts. Genuine question : is there something fundamentally wrong with Ollama ? I use Ollama because it is easy to set up and manage (and also because VLLM is not super Windows friendly). I thought the main advantage of VLLM was better concurrency management (better batching). But if the quality of the inte", "context": "", "use_case": "LLM inference serving", "switched_from": "", "story": "Why your local LLM feels dumber than it is"}, {"id": 736557456, "item_id": 46034841, "url": "https://news.ycombinator.com/item?id=46034841", "by": "Yarden_Bruch_El", "date": "2025-11-24", "stance": "neutral", "quote": "We built a custom inference layer on top of vLLM", "excerpt": "Hi HN, We built a platform for orchestrating multi-agent debates (e.g., \"Security\" vs. \"Refactoring\" experts). The Challenge: Standard sequential agent chains (A -> B -> C) are too slow for real-time chat. The Fix (vLLM): We built a custom inference layer on top of vLLM to solve the bottleneck: Parallelism: We use continuous batching to generate multiple agent responses simultaneously rather than waiting for sequenti", "context": "", "use_case": "parallel multi-agent inference", "switched_from": "", "story": "BYO \u2013 A multi-agent runtime optimized for parallel inference"}, {"id": 777398657, "item_id": 48587416, "url": "https://news.ycombinator.com/item?id=48587416", "by": "alexellisuk", "date": "2026-06-18", "stance": "compare", "quote": "We did run vLLM on the 3090s \u2014 measured ~3 tok/s slower on generation for our single-to-few-user pattern", "excerpt": "We did run vLLM on the 3090s \u2014 measured ~3 tok/s slower on generation for our single-to-few-user pattern, plus less flexibility on quant and slower startup (actual minutes vs single digit seconds). We may do more with it again in the future - there isn't unlimited time for us to tinker, I'm sharing our journey (so far) and reasoning. It's the right call for concurrent batched serving (barrkel's point downthread is sp", "context": "", "use_case": "LLM serving", "switched_from": "", "story": "Local Qwen isn't a worse Opus, it's a different tool"}]}, {"text": " and for production inference, citing broad ecosystem support.", "cites": [{"id": 785962608, "item_id": 49122663, "url": "https://news.ycombinator.com/item?id=49122663", "by": "rurban", "date": "2026-07-31", "stance": "recommend", "quote": "For LLM inference in production we use vLLM (with our own model).", "excerpt": "For LLM inference in production we use vLLM (with our own model).", "context": "", "use_case": "LLM inference in production", "switched_from": "", "story": "Ask HN: What are you using for LLM inference in production?"}, {"id": 784353520, "item_id": 49022095, "url": "https://news.ycombinator.com/item?id=49022095", "by": "scottcha", "date": "2026-07-23", "stance": "recommend", "quote": "We use vllm as it generally has the best ecosystem support.", "excerpt": "We use vllm as it generally has the best ecosystem support. Parameters are largely dependent on what type of requests you are serving (concurrency, input/output ratios, cached hit patterns). We've never had a limitation at the tokenizer step. Limitations at peak tend to manifest more on slower time in vllm doing prefill or decode though we actively try and minimize this.", "context": "", "use_case": "LLM serving", "switched_from": "", "story": "GigaToken: ~1000x faster Language model tokenization"}, {"id": 790556496, "item_id": 49409781, "url": "https://news.ycombinator.com/item?id=49409781", "by": "TofuLover", "date": "2026-08-23", "stance": "recommend", "quote": "vLLM gang all the way!", "excerpt": "These are great questions, thanks! I'll answer them in turn, in a list because I like lists. * Confidentiality: no logging, third party analytics, or anything like that. More details in our Privacy Poilicy [1]. Our hosting providers will have their own policies, but we're not running a super private service like Proton or similar. Might do some kind of secure tenancy in the future if there's demand. * Price: I think ", "context": "runs an open-model API service", "use_case": "serving open models", "switched_from": "", "story": "I gave Qwen 3.8 27B a reverse-engineering job and it finished in 30 minutes"}]}, {"text": " The most common complaints are a hard, tweak-heavy setup", "cites": [{"id": 732496465, "item_id": 45781029, "url": "https://news.ycombinator.com/item?id=45781029", "by": "embedding-shape", "date": "2025-11-01", "stance": "recommend", "quote": "Need batching + highest total throughoutput? vLLM, complicated to deploy and install though", "excerpt": "Not parent, but frequent user of GPT-OSS, tried all different ways of running it. Choice goes something like this: - Need batching + highest total throughoutput? vLLM, complicated to deploy and install though, need special versions for top performance with GPT-OSS - Easiest to manage + fast enough: llama.cpp, easier to deploy as well (just a binary) and super fast, getting ~260 tok/s on a RTX Pro 6000 for the 20B ver", "context": "", "use_case": "high-throughput batched serving", "switched_from": "", "story": "Ask HN: Who uses open LLMs and coding assistants locally? Share setup and laptop"}, {"id": 793011682, "item_id": 49563230, "url": "https://news.ycombinator.com/item?id=49563230", "by": "KronisLV", "date": "2026-09-04", "stance": "recommend", "quote": "there's llama.cpp and vLLM, each harder to setup but opens the door for squeezing more performance out of your hardware", "excerpt": "> Download ollama and give it a try Weren't people pissed off at them for a variety of reasons? Not giving attribution etc. In case anyone wants other good alternatives, there's llama.cpp and vLLM, each harder to setup but opens the door for squeezing more performance out of your hardware. I've also heard okay things about LM Studio and I think Unsloth had their own thing as well: https://unsloth.ai/docs/new/studio >", "context": "", "use_case": "running local LLMs", "switched_from": "", "story": "Which tools do Claude, Codex and Cursor choose? We measured 17k runs to find out"}, {"id": 779940113, "item_id": 48746257, "url": "https://news.ycombinator.com/item?id=48746257", "by": "coder543", "date": "2026-07-01", "stance": "warn", "quote": "vLLM is very painful, even with the Spark community docker image.", "excerpt": "It\u2019s not FUD. It is my actual, lived experience. FUD is false, which this is not. I use both vLLM and llama-server. vLLM is very painful, even with the Spark community docker image. It is slow to start, it does not support 3-bit dynamic quants well, and it takes a lot of tweaking to get it to run well for each model I want to try out, which is made worse by the slow starts. I\u2019m glad you\u2019ve had a better experience? I ", "context": "runs local models on a Spark", "use_case": "local LLM inference", "switched_from": "", "story": "Qwen 3.6 27B is the sweet spot for local development"}, {"id": 746904496, "item_id": 46681531, "url": "https://news.ycombinator.com/item?id=46681531", "by": "jakeasmith", "date": "2026-01-19", "stance": "neutral", "quote": "I wanted to experiment with running something a little more \"low level\".", "excerpt": "I've been running Ollama in my home lab for awhile now but I wanted to experiment with running something a little more \"low level\". I saw a ton of posts about Llama.cpp, which looked interesting, but there wasn't a lot specifically about running vLLM on Proxmox, so I thought I'd give it a try. Setting up vLLM in an LXC wasn\u2019t necessarily difficult but, even after doing it several times, it was still tedious. These ar", "context": "home lab user", "use_case": "LLM serving in Proxmox LXC", "switched_from": "Ollama", "story": "Running a vLLM LXC on Proxmox 9 with Nvidia GPU Passthrough"}]}, {"text": " and frustrating bugs.", "cites": [{"id": 791889696, "item_id": 49493106, "url": "https://news.ycombinator.com/item?id=49493106", "by": "kouteiheika", "date": "2026-08-29", "stance": "warn", "quote": "I love vLLM, but damn if it isn't frustratingly buggy.", "excerpt": "I love vLLM, but damn if it isn't frustratingly buggy. I was recently running DeepSeek-V4-Flash on a B300. On v0.26 it was totally broken, and I had to add three out-of-tree patches to fix it. I updated to v0.27 -- no patches necessary now, but the output is now broken as it randomly starts responding with garbage (repeated token loops). On my workstation where I run Gemma-4 on an RTX 6000 the whole process tends to ", "context": "Runs DeepSeek-V4-Flash on a B300 and Gemma-4 on an RTX 6000", "use_case": "serving LLMs on GPUs", "switched_from": "", "story": "vLLM v0.28.0"}, {"id": 791896689, "item_id": 49493543, "url": "https://news.ycombinator.com/item?id=49493543", "by": "bugglebeetle", "date": "2026-08-29", "stance": "warn", "quote": "vLLM seems to have a lot more gotchas and kludged together stuff", "excerpt": "I recently switched to using for some large volume inference with small, fine-tuned models and was surprised how nice and painless things were. vLLM seems to have a lot more gotchas and kludged together stuff once you get outside of anything straightforward.", "context": "", "use_case": "LLM inference", "switched_from": "", "story": "vLLM v0.28.0"}, {"id": 791902592, "item_id": 49493912, "url": "https://news.ycombinator.com/item?id=49493912", "by": "joshheitzman", "date": "2026-08-29", "stance": "warn", "quote": "I was hoping to see the reasoning_content mess get robustly fixed", "excerpt": "I was hoping to see the reasoning_content mess get robustly fixed, but all we got was this doc change: https://github.com/vllm-project/vllm/pull/50624", "context": "", "use_case": "LLM inference", "switched_from": "", "story": "vLLM v0.28.0"}]}], "praise": [{"label": "Batched throughput", "summary": "Commenters pick it for concurrent, batched serving and for getting more performance out of hardware.", "quotes": [{"id": 732496465, "item_id": 45781029, "url": "https://news.ycombinator.com/item?id=45781029", "by": "embedding-shape", "date": "2025-11-01", "stance": "recommend", "quote": "Need batching + highest total throughoutput? vLLM, complicated to deploy and install though", "excerpt": "Not parent, but frequent user of GPT-OSS, tried all different ways of running it. Choice goes something like this: - Need batching + highest total throughoutput? vLLM, complicated to deploy and install though, need special versions for top performance with GPT-OSS - Easiest to manage + fast enough: llama.cpp, easier to deploy as well (just a binary) and super fast, getting ~260 tok/s on a RTX Pro 6000 for the 20B ver", "context": "", "use_case": "high-throughput batched serving", "switched_from": "", "story": "Ask HN: Who uses open LLMs and coding assistants locally? Share setup and laptop"}, {"id": 790464065, "item_id": 49404004, "url": "https://news.ycombinator.com/item?id=49404004", "by": "JacobJack", "date": "2026-08-22", "stance": "compare", "quote": "I thought the main advantage of VLLM was better concurrency management (better batching).", "excerpt": "> And the comparisons in this post are not going to be running some 2.58-bit-gguf-in-ollama with a couple test prompts. Genuine question : is there something fundamentally wrong with Ollama ? I use Ollama because it is easy to set up and manage (and also because VLLM is not super Windows friendly). I thought the main advantage of VLLM was better concurrency management (better batching). But if the quality of the inte", "context": "", "use_case": "LLM inference serving", "switched_from": "", "story": "Why your local LLM feels dumber than it is"}, {"id": 736557456, "item_id": 46034841, "url": "https://news.ycombinator.com/item?id=46034841", "by": "Yarden_Bruch_El", "date": "2025-11-24", "stance": "neutral", "quote": "We built a custom inference layer on top of vLLM", "excerpt": "Hi HN, We built a platform for orchestrating multi-agent debates (e.g., \"Security\" vs. \"Refactoring\" experts). The Challenge: Standard sequential agent chains (A -> B -> C) are too slow for real-time chat. The Fix (vLLM): We built a custom inference layer on top of vLLM to solve the bottleneck: Parallelism: We use continuous batching to generate multiple agent responses simultaneously rather than waiting for sequenti", "context": "", "use_case": "parallel multi-agent inference", "switched_from": "", "story": "BYO \u2013 A multi-agent runtime optimized for parallel inference"}]}, {"label": "Production use", "summary": "Teams run it for production inference and cite its ecosystem support.", "quotes": [{"id": 785962608, "item_id": 49122663, "url": "https://news.ycombinator.com/item?id=49122663", "by": "rurban", "date": "2026-07-31", "stance": "recommend", "quote": "For LLM inference in production we use vLLM (with our own model).", "excerpt": "For LLM inference in production we use vLLM (with our own model).", "context": "", "use_case": "LLM inference in production", "switched_from": "", "story": "Ask HN: What are you using for LLM inference in production?"}, {"id": 784353520, "item_id": 49022095, "url": "https://news.ycombinator.com/item?id=49022095", "by": "scottcha", "date": "2026-07-23", "stance": "recommend", "quote": "We use vllm as it generally has the best ecosystem support.", "excerpt": "We use vllm as it generally has the best ecosystem support. Parameters are largely dependent on what type of requests you are serving (concurrency, input/output ratios, cached hit patterns). We've never had a limitation at the tokenizer step. Limitations at peak tend to manifest more on slower time in vllm doing prefill or decode though we actively try and minimize this.", "context": "", "use_case": "LLM serving", "switched_from": "", "story": "GigaToken: ~1000x faster Language model tokenization"}, {"id": 790556496, "item_id": 49409781, "url": "https://news.ycombinator.com/item?id=49409781", "by": "TofuLover", "date": "2026-08-23", "stance": "recommend", "quote": "vLLM gang all the way!", "excerpt": "These are great questions, thanks! I'll answer them in turn, in a list because I like lists. * Confidentiality: no logging, third party analytics, or anything like that. More details in our Privacy Poilicy [1]. Our hosting providers will have their own policies, but we're not running a super private service like Proton or similar. Might do some kind of secure tenancy in the future if there's demand. * Price: I think ", "context": "runs an open-model API service", "use_case": "serving open models", "switched_from": "", "story": "I gave Qwen 3.8 27B a reverse-engineering job and it finished in 30 minutes"}]}, {"label": "Cloud deployment", "summary": "Commenters run it on rented cloud GPUs or through a Kubernetes operator.", "quotes": [{"id": 782166928, "item_id": 48885433, "url": "https://news.ycombinator.com/item?id=48885433", "by": "blinded", "date": "2026-07-12", "stance": "recommend", "quote": "I spin up a gpu instance in a cloud, run my model via vllm, connect to it via an ssh tunnel. done.", "excerpt": "I spin up a gpu instance in a cloud, run my model via vllm, connect to it via an ssh tunnel. done.", "context": "", "use_case": "private LLM serving", "switched_from": "", "story": "Ask HN: How do you use LLMs for private discussions?"}, {"id": 782174896, "item_id": 48885931, "url": "https://news.ycombinator.com/item?id=48885931", "by": "blinded", "date": "2026-07-12", "stance": "neutral", "quote": "you need to run vllm and a model", "excerpt": "Hey sure. It depends, but usually spin up an h100 on lambda.ai or coreweave. They have capacity and their UIs/APIs are nice. I spin it up for an hour or two, believe it was 6~ dollars an hour. Once the gpu instance is up, you need to run vllm and a model, ie https://docs.lambda.ai/education/large-language-models/deploying-nemotron-3-nano/. Then you can connect your pi.dev, openwebui, etc etc to vllm and interact with", "context": "", "use_case": "serving private LLM on rented GPU", "switched_from": "", "story": "Ask HN: How do you use LLMs for private discussions?"}, {"id": 775778243, "item_id": 48486140, "url": "https://news.ycombinator.com/item?id=48486140", "by": "stevefan1999", "date": "2026-06-10", "stance": "recommend", "quote": "using just the ollama, llama.cpp containers and especially vllm operator", "excerpt": "> Ever hear of a large K8s migration going well ...well, AI infra stacks are more or less k8s/cloud native, especially on the inference side. Nvidia GPU operator plus KubeFlow makes deploying models easy, and I did that manually as well using just the ollama, llama.cpp containers and especially vllm operator. I'm not sure about what's the measuring notation of \"large\" is, but OpenAI had a blog post about how to manag", "context": "", "use_case": "model serving", "switched_from": "", "story": "Claude Desktop spins up a VM without no way of stopping it"}]}, {"label": "Prompt caching", "summary": "Some users switched from llama.cpp to avoid its prompt cache bugs with Qwen and Gemma models.", "quotes": [{"id": 774215824, "item_id": 48388489, "url": "https://news.ycombinator.com/item?id=48388489", "by": "verdverm", "date": "2026-06-03", "stance": "recommend", "quote": "I switched from llama.cpp to vLLM because of prompt cache bugs in qwen/gemma models", "excerpt": "I switched from llama.cpp to vLLM because of prompt cache bugs in qwen/gemma models This is a good starting issue with a bunch of linked/related https://github.com/ggml-org/llama.cpp/issues/22746", "context": "", "use_case": "local model inference", "switched_from": "llama.cpp", "story": "Gemma 4 12B: A unified, encoder-free multimodal model"}, {"id": 773231073, "item_id": 48326942, "url": "https://news.ycombinator.com/item?id=48326942", "by": "verdverm", "date": "2026-05-29", "stance": "recommend", "quote": "OpenCode + vllm, model will depend on your hardware", "excerpt": "OpenCode + vllm, model will depend on your hardware, but OpenCode also has a killer $10/m plan with quotas for some top tier open weight models. I'm using qwen3.6 on a DGX spark, llama-cpp has prompt cache bugs for qwen/gemma models (among more being reported). Using my OpenCode-go sub when I want a bigger / more capable model", "context": "runs models on a DGX Spark", "use_case": "local model serving", "switched_from": "", "story": "Ask HN: If I cancel Codex today whats the next best local inference agent?"}]}, {"label": "Quantization options", "summary": "Commenters point to quantization support such as turboquant and third-party libraries like glq.", "quotes": [{"id": 766015360, "item_id": 47875960, "url": "https://news.ycombinator.com/item?id=47875960", "by": "AlexC04", "date": "2026-04-23", "stance": "recommend", "quote": "vLLM apparently already has an implementation of turboquant available", "excerpt": "One other thing you might want to check out for running locally. (I have not independently verified yet, it's on the TODO list though) https://docs.vllm.ai/en/latest/api/vllm/model_executor/layers/quantization/turboquant/ vLLM apparently already has an implementation of turboquant available - which is said to losslessly reduce the memory footprint required by 6x and improve inference speed by 8x. From what I understa", "context": "developer-tools project, two young kids", "use_case": "local LLM inference", "switched_from": "", "story": "Qwen3.6-27B: Flagship-Level Coding in a 27B Dense Model"}, {"id": 789605057, "item_id": 49350316, "url": "https://news.ycombinator.com/item?id=49350316", "by": "acd", "date": "2026-08-18", "stance": "neutral", "quote": "Glq runs with vLLM.", "excerpt": "I have written a llm compression quantization library called glq because of the high ram prices. Glq uses qtip Trellis quantization which are efficient at low bpw 2-4 bits. As a gamer and ai developer I want to squeeze more out of the same hardware. Glq runs with vLLM. Open source https://github.com/cnygaard/glq Pypi glq https://pypi.org/project/glq/", "context": "", "use_case": "LLM inference", "switched_from": "", "story": "Memory prices climb 500% in 12 months, up to 10x the lowest ever tracked prices"}]}], "caveats": [{"label": "Setup effort", "summary": "Installation and per-model configuration are described as complicated or tedious.", "quotes": [{"id": 732496465, "item_id": 45781029, "url": "https://news.ycombinator.com/item?id=45781029", "by": "embedding-shape", "date": "2025-11-01", "stance": "recommend", "quote": "Need batching + highest total throughoutput? vLLM, complicated to deploy and install though", "excerpt": "Not parent, but frequent user of GPT-OSS, tried all different ways of running it. Choice goes something like this: - Need batching + highest total throughoutput? vLLM, complicated to deploy and install though, need special versions for top performance with GPT-OSS - Easiest to manage + fast enough: llama.cpp, easier to deploy as well (just a binary) and super fast, getting ~260 tok/s on a RTX Pro 6000 for the 20B ver", "context": "", "use_case": "high-throughput batched serving", "switched_from": "", "story": "Ask HN: Who uses open LLMs and coding assistants locally? Share setup and laptop"}, {"id": 779940113, "item_id": 48746257, "url": "https://news.ycombinator.com/item?id=48746257", "by": "coder543", "date": "2026-07-01", "stance": "warn", "quote": "vLLM is very painful, even with the Spark community docker image.", "excerpt": "It\u2019s not FUD. It is my actual, lived experience. FUD is false, which this is not. I use both vLLM and llama-server. vLLM is very painful, even with the Spark community docker image. It is slow to start, it does not support 3-bit dynamic quants well, and it takes a lot of tweaking to get it to run well for each model I want to try out, which is made worse by the slow starts. I\u2019m glad you\u2019ve had a better experience? I ", "context": "runs local models on a Spark", "use_case": "local LLM inference", "switched_from": "", "story": "Qwen 3.6 27B is the sweet spot for local development"}, {"id": 746904496, "item_id": 46681531, "url": "https://news.ycombinator.com/item?id=46681531", "by": "jakeasmith", "date": "2026-01-19", "stance": "neutral", "quote": "I wanted to experiment with running something a little more \"low level\".", "excerpt": "I've been running Ollama in my home lab for awhile now but I wanted to experiment with running something a little more \"low level\". I saw a ton of posts about Llama.cpp, which looked interesting, but there wasn't a lot specifically about running vLLM on Proxmox, so I thought I'd give it a try. Setting up vLLM in an LXC wasn\u2019t necessarily difficult but, even after doing it several times, it was still tedious. These ar", "context": "home lab user", "use_case": "LLM serving in Proxmox LXC", "switched_from": "Ollama", "story": "Running a vLLM LXC on Proxmox 9 with Nvidia GPU Passthrough"}]}, {"label": "Bugs", "summary": "Users report broken releases, garbage output, hangs, and messy reasoning_content handling.", "quotes": [{"id": 791889696, "item_id": 49493106, "url": "https://news.ycombinator.com/item?id=49493106", "by": "kouteiheika", "date": "2026-08-29", "stance": "warn", "quote": "I love vLLM, but damn if it isn't frustratingly buggy.", "excerpt": "I love vLLM, but damn if it isn't frustratingly buggy. I was recently running DeepSeek-V4-Flash on a B300. On v0.26 it was totally broken, and I had to add three out-of-tree patches to fix it. I updated to v0.27 -- no patches necessary now, but the output is now broken as it randomly starts responding with garbage (repeated token loops). On my workstation where I run Gemma-4 on an RTX 6000 the whole process tends to ", "context": "Runs DeepSeek-V4-Flash on a B300 and Gemma-4 on an RTX 6000", "use_case": "serving LLMs on GPUs", "switched_from": "", "story": "vLLM v0.28.0"}, {"id": 791896689, "item_id": 49493543, "url": "https://news.ycombinator.com/item?id=49493543", "by": "bugglebeetle", "date": "2026-08-29", "stance": "warn", "quote": "vLLM seems to have a lot more gotchas and kludged together stuff", "excerpt": "I recently switched to using for some large volume inference with small, fine-tuned models and was surprised how nice and painless things were. vLLM seems to have a lot more gotchas and kludged together stuff once you get outside of anything straightforward.", "context": "", "use_case": "LLM inference", "switched_from": "", "story": "vLLM v0.28.0"}, {"id": 791902592, "item_id": 49493912, "url": "https://news.ycombinator.com/item?id=49493912", "by": "joshheitzman", "date": "2026-08-29", "stance": "warn", "quote": "I was hoping to see the reasoning_content mess get robustly fixed", "excerpt": "I was hoping to see the reasoning_content mess get robustly fixed, but all we got was this doc change: https://github.com/vllm-project/vllm/pull/50624", "context": "", "use_case": "LLM inference", "switched_from": "", "story": "vLLM v0.28.0"}]}, {"label": "Startup and latency", "summary": "Startup is slow, and one team measured slower generation for single-to-few-user workloads.", "quotes": [{"id": 779940113, "item_id": 48746257, "url": "https://news.ycombinator.com/item?id=48746257", "by": "coder543", "date": "2026-07-01", "stance": "warn", "quote": "vLLM is very painful, even with the Spark community docker image.", "excerpt": "It\u2019s not FUD. It is my actual, lived experience. FUD is false, which this is not. I use both vLLM and llama-server. vLLM is very painful, even with the Spark community docker image. It is slow to start, it does not support 3-bit dynamic quants well, and it takes a lot of tweaking to get it to run well for each model I want to try out, which is made worse by the slow starts. I\u2019m glad you\u2019ve had a better experience? I ", "context": "runs local models on a Spark", "use_case": "local LLM inference", "switched_from": "", "story": "Qwen 3.6 27B is the sweet spot for local development"}, {"id": 777398657, "item_id": 48587416, "url": "https://news.ycombinator.com/item?id=48587416", "by": "alexellisuk", "date": "2026-06-18", "stance": "compare", "quote": "We did run vLLM on the 3090s \u2014 measured ~3 tok/s slower on generation for our single-to-few-user pattern", "excerpt": "We did run vLLM on the 3090s \u2014 measured ~3 tok/s slower on generation for our single-to-few-user pattern, plus less flexibility on quant and slower startup (actual minutes vs single digit seconds). We may do more with it again in the future - there isn't unlimited time for us to tinker, I'm sharing our journey (so far) and reasoning. It's the right call for concurrent batched serving (barrkel's point downthread is sp", "context": "", "use_case": "LLM serving", "switched_from": "", "story": "Local Qwen isn't a worse Opus, it's a different tool"}]}, {"label": "Quant formats", "summary": "GGUF and 3-bit dynamic quant support are weak, and sharing weights with llama.cpp needs conversion.", "quotes": [{"id": 755129489, "item_id": 47195593, "url": "https://news.ycombinator.com/item?id=47195593", "by": "deepsquirrelnet", "date": "2026-02-28", "stance": "neutral", "quote": "I only wish gguf format had better vllm support.", "excerpt": "I love the work unsloth is doing. I only wish gguf format had better vllm support. It\u2019s sometimes hard to find trustworthy quants that work well with vllm.", "context": "", "use_case": "model serving", "switched_from": "", "story": "Unsloth Dynamic 2.0 GGUFs"}, {"id": 779940113, "item_id": 48746257, "url": "https://news.ycombinator.com/item?id=48746257", "by": "coder543", "date": "2026-07-01", "stance": "warn", "quote": "vLLM is very painful, even with the Spark community docker image.", "excerpt": "It\u2019s not FUD. It is my actual, lived experience. FUD is false, which this is not. I use both vLLM and llama-server. vLLM is very painful, even with the Spark community docker image. It is slow to start, it does not support 3-bit dynamic quants well, and it takes a lot of tweaking to get it to run well for each model I want to try out, which is made worse by the slow starts. I\u2019m glad you\u2019ve had a better experience? I ", "context": "runs local models on a Spark", "use_case": "local LLM inference", "switched_from": "", "story": "Qwen 3.6 27B is the sweet spot for local development"}, {"id": 788312928, "item_id": 49269558, "url": "https://news.ycombinator.com/item?id=49269558", "by": "embedding-shape", "date": "2026-08-12", "stance": "neutral", "quote": "I mostly end up using vLLM these days", "excerpt": "> At this point the options are llama-server or vLLM Which last time I checked, both use different formats of the weights, the former GGUF while the latter .safetensors. I mostly end up using vLLM these days and I'm a bit more performance sensitive than what I used to be. Just a shame it's a hassle to share the weights between them with conversion and what not, either batched or on-startup.", "context": "performance sensitive user", "use_case": "running local models", "switched_from": "", "story": "llama.cpp"}]}, {"label": "Dependency builds", "summary": "One user could not build the container with updated dependencies and switched to llama.cpp.", "quotes": [{"id": 768907056, "item_id": 48056691, "url": "https://news.ycombinator.com/item?id=48056691", "by": "cookiengineer", "date": "2026-05-07", "stance": "warn", "quote": "You still can't build the vllm container with updated dependencies since llmlite got pwned.", "excerpt": "Fun fact: You still can't build the vllm container with updated dependencies since llmlite got pwned. Either due to regression bugs, or due to impossible transient dependencies in the dependency tree that are not resolvable. There is just too much slopcode down the line, and too many dependencies relying on pinned outdated (and unpublished) dependencies. I switched to llama.cpp because of that. To me it feels more an", "context": "", "use_case": "LLM inference", "switched_from": "", "story": "Maybe you shouldn't install new software for a bit"}]}, {"label": "GPU memory", "summary": "One RTX 5090 owner ran out of GPU memory.", "quotes": [{"id": 788973504, "item_id": 49310844, "url": "https://news.ycombinator.com/item?id=49310844", "by": "sgt", "date": "2026-08-15", "stance": "warn", "quote": "I already tried using vLLM but it ran out of GPU memory.", "excerpt": "Amazing. I'll give it a shot on my 5090. I already tried using vLLM but it ran out of GPU memory. I guess it's likely Llama.cpp will work.", "context": "RTX 5090 owner", "use_case": "running Qwen locally", "switched_from": "", "story": "Qwen 3.8 27B is out: open weights, best local dense model yet"}]}], "who": [{"by": "verdverm", "context": "runs vLLM on k8s", "url": "https://news.ycombinator.com/item?id=49793839", "date": "2026-09-21"}, {"by": "mmastrac", "context": "runs on a DGX Spark", "url": "https://news.ycombinator.com/item?id=49755873", "date": "2026-09-18"}, {"by": "TofuLover", "context": "runs an open-model API service", "url": "https://news.ycombinator.com/item?id=49409781", "date": "2026-08-23"}, {"by": "embedding-shape", "context": "performance sensitive user", "url": "https://news.ycombinator.com/item?id=49269558", "date": "2026-08-12"}, {"by": "pet_the_bird", "context": "has access to ~320 GB memory", "url": "https://news.ycombinator.com/item?id=48619010", "date": "2026-06-21"}, {"by": "AlexC04", "context": "developer-tools project, two young kids", "url": "https://news.ycombinator.com/item?id=47875960", "date": "2026-04-23"}, {"by": "jakeasmith", "context": "home lab user", "url": "https://news.ycombinator.com/item?id=46681531", "date": "2026-01-19"}], "tags": ["for production serving", "self-hostable", "for cloud GPU instances", "for Kubernetes", "for local LLMs"], "use_cases": ["llm inference", "model serving", "llm inference serving", "llm serving", "local llm inference", "serving models"], "compared": [{"slug": "llama-cpp", "name": "llama.cpp", "co": 17, "switched_from": 1, "switched_to": 2}, {"slug": "ollama", "name": "Ollama", "co": 7, "switched_from": 1, "switched_to": 0}, {"slug": "sglang", "name": "SGLang", "co": 3, "switched_from": 0, "switched_to": 1}, {"slug": "llama-server", "name": "llama-server", "co": 2, "switched_from": 0, "switched_to": 1}, {"slug": "codex", "name": "Codex", "co": 3, "switched_from": 0, "switched_to": 0}, {"slug": "pi", "name": "Pi", "co": 3, "switched_from": 0, "switched_to": 0}, {"slug": "qwen-3-6", "name": "Qwen 3.6", "co": 3, "switched_from": 0, "switched_to": 0}, {"slug": "opencode", "name": "OpenCode", "co": 2, "switched_from": 0, "switched_to": 0}], "makers": [{"id": 803917168, "item_id": 50244823, "url": "https://news.ycombinator.com/item?id=50244823", "by": "mmastrac", "date": "2026-10-26", "stance": "neutral", "quote": "I worked on the vLLM feature", "excerpt": "Amusingly constrained decoding is the same approach we use to turn an AR model into a /v1/systemone endpoint (I worked on the vLLM feature). Tool calling is fraught with peril, especially because there's nothing that necessarily says that a models tools are fixed in place for the life of its run. In some cases, tools are added mid-run to avoid a giant prefix miss and (at least last time I checked!), the industry has ", "context": "worked on a vLLM feature", "use_case": "constrained decoding for tool calls", "switched_from": "", "story": "Fixing Tool-Call Accuracy in vLLM"}, {"id": 799307201, "item_id": 49956700, "url": "https://news.ycombinator.com/item?id=49956700", "by": "amankarki", "date": "2026-10-04", "stance": "neutral", "quote": "now working on block-FP8 GEMM support for Ada GPUs (L4/L40S/4090) in vLLM", "excerpt": "Location: India (IST, GMT+5:30) Remote: Yes Willing to relocate: No Technologies: C++20, CUDA, CUTLASS, Triton, Python, PyTorch, vLLM, llama.cpp/ggml, FP8/INT8 quantization, Linux R\u00e9sum\u00e9/CV: https://aman-portfolio-rho-six.vercel.app/resume.pdf Email: itsamankarki@gmail.com I write C++ and CUDA for LLM inference. 2 CUDA PRs merged into llama.cpp; now working on block-FP8 GEMM support for Ada GPUs (L4/L40S/4090) in vLL", "context": "C++/CUDA engineer for LLM inference", "use_case": "FP8 GEMM support", "switched_from": "", "story": "Ask HN: Who wants to be hired? (October 2026)"}], "threads": [{"url": "https://news.ycombinator.com/item?id=49492067", "title": "vLLM v0.28.0", "mentions": 3}, {"url": "https://news.ycombinator.com/item?id=49299605", "title": "Qwen 3.8 27B is out: open weights, best local dense model yet", "mentions": 2}, {"url": "https://news.ycombinator.com/item?id=48885422", "title": "Ask HN: How do you use LLMs for private discussions?", "mentions": 2}, {"url": "https://news.ycombinator.com/item?id=50242870", "title": "Fixing Tool-Call Accuracy in vLLM", "mentions": 1}, {"url": "https://news.ycombinator.com/item?id=49922568", "title": "Ask HN: Who wants to be hired? (October 2026)", "mentions": 1}, {"url": "https://news.ycombinator.com/item?id=49911997", "title": "Lichen \u2013 A local, BOY model system1 (Jev) server with image support", "mentions": 1}], "sentiment_by_month": {"2025-11": {"recommend": 1, "neutral": 1}, "2025-12": {}, "2026-01": {"neutral": 1, "compare": 1}, "2026-02": {"neutral": 2}, "2026-03": {}, "2026-04": {"recommend": 1}, "2026-05": {"warn": 1, "recommend": 1}, "2026-06": {"recommend": 2, "compare": 2, "neutral": 1}, "2026-07": {"warn": 1, "recommend": 3, "neutral": 2}, "2026-08": {"neutral": 5, "warn": 4, "compare": 1, "recommend": 1}, "2026-09": {"recommend": 2, "neutral": 3}, "2026-10": {}}, "quotes": [{"id": 798591970, "item_id": 49911998, "url": "https://news.ycombinator.com/item?id=49911998", "by": "jeff_ciesielski", "date": "2026-09-30", "stance": "neutral", "quote": "Served from vLLM with the NVFP4 build, it scores 204", "excerpt": "Howdy folks. I was inspired by the really neat \"Jev in 25 lines of Python\" post a few days ago (and System One models in general), and after playing with the idea I tried whipping up something a little more complex, and Lichen is what popped out: https://github.com/Mushroom-Systems/lichen My question after reading that post (and I know I wasn't the only one; antirez came right out and said it in the comments) was whe", "context": "", "use_case": "serving models", "switched_from": "", "story": "Lichen \u2013 A local, BOY model system1 (Jev) server with image support"}, {"id": 798446131, "item_id": 49902883, "url": "https://news.ycombinator.com/item?id=49902883", "by": "latentsea", "date": "2026-09-30", "stance": "recommend", "quote": "A dual R9700 rig is really the sweet spot right now with the vLLM-radiance fork.", "excerpt": "Qwen3.8-27B is a huge step up from Qwen3.6-27B. That release only happened relatively recently but that felt like the 'Opus 4.5' release turning point that SOTA models experienced back when that came out. It was the first time I felt like local models are actually good enough to use as daily drivers now. So, it was only after that point that I switched. Qwen3.8-Flash-Next is better still if you can run fast enough. I", "context": "", "use_case": "local inference", "switched_from": "", "story": "GPT 6.1 Sol: Near-Astra intelligence for a fifth of the price"}, {"id": 796841744, "item_id": 49802609, "url": "https://news.ycombinator.com/item?id=49802609", "by": "verdverm", "date": "2026-09-22", "stance": "neutral", "quote": "vLLM has a PR very close to merging", "excerpt": "With all the excitement around Jev, I suspect we'll see hundreds of options, it doesn't sound like Jev is that hard to replicate, given all the 3rd parties who are getting pretty damn close, or even better, results within a week. vLLM has a PR very close to merging: https://github.com/vllm-project/vllm/pull/57250 Kev is an open Jev: https://github.com/jaredpalmer/kev", "context": "", "use_case": "LLM inference", "switched_from": "", "story": "OpenAI is well positioned to fast-follow Jev"}, {"id": 796732272, "item_id": 49795767, "url": "https://news.ycombinator.com/item?id=49795767", "by": "verdverm", "date": "2026-09-22", "stance": "neutral", "quote": "vLLM PR nearing merge readiness for Jev like capabilities with diffusion models", "excerpt": "related, vLLM PR nearing merge readiness for Jev like capabilities with diffusion models https://github.com/vllm-project/vllm/pull/57250", "context": "", "use_case": "LLM inference serving", "switched_from": "", "story": "Kev: Tiny Jev-like family of decision models built on top of Qwen3.5"}, {"id": 796701424, "item_id": 49793839, "url": "https://news.ycombinator.com/item?id=49793839", "by": "verdverm", "date": "2026-09-21", "stance": "neutral", "quote": "I have a quirky vLLM on k8s on 2x OEM sparks setup with 9 models available to me.", "excerpt": "I have a quirky vLLM on k8s on 2x OEM sparks setup with 9 models available to me. I'm not keen to run nightly vLLM, too many issues with it in the past. Going the qwen-next path means displacing things I use daily :/ I have a watchful eye on the diffusion ~ Jev/Kev PR https://github.com/vllm-project/vllm/pull/57250", "context": "runs vLLM on k8s", "use_case": "model serving", "switched_from": "", "story": "MiMo v2.6"}, {"id": 796093968, "item_id": 49755873, "url": "https://news.ycombinator.com/item?id=49755873", "by": "mmastrac", "date": "2026-09-18", "stance": "neutral", "quote": "the vLLM patch to turn DiffusionGemma into Jev is available", "excerpt": "If you want to try a _legit_ Jev implementation that matches (at least in my evals), the vLLM patch to turn DiffusionGemma into Jev is available. On my DGX Spark I get very similar latency numbers, and it matches my evals + or - a few points on each test (DG wins some, Jev wins some, both show low confidence when wrong). I ran the same evals against a Qwen36 and it clearly lost to both of them, so you are leaving bot", "context": "runs on a DGX Spark", "use_case": "serving Jev-style models", "switched_from": "", "story": "OpenJev"}, {"id": 794202774, "item_id": 49637673, "url": "https://news.ycombinator.com/item?id=49637673", "by": "softwarewright", "date": "2026-09-10", "stance": "neutral", "quote": "served by llama.cpp, Freetoken, or vLLM", "excerpt": "Claude Code, Codex, opencode (Z.ai GLM-5.x), sometimes Gemini -- most of my work is at Mac iTerm2 CLI or Arch Linux terminal. I sometimes use emacs as the coding agent, but rarely JetBrains IDE or vscode (more for reading/debugging code). I supplement this with my own vibe-coded tools that help agents plan, perform sagas/steps, stay on track, tools that check the code produced, tools that check the produced documenta", "context": "", "use_case": "serving local LLMs", "switched_from": "", "story": "Ask HN: What's your AI coding set up?"}, {"id": 793011682, "item_id": 49563230, "url": "https://news.ycombinator.com/item?id=49563230", "by": "KronisLV", "date": "2026-09-04", "stance": "recommend", "quote": "there's llama.cpp and vLLM, each harder to setup but opens the door for squeezing more performance out of your hardware", "excerpt": "> Download ollama and give it a try Weren't people pissed off at them for a variety of reasons? Not giving attribution etc. In case anyone wants other good alternatives, there's llama.cpp and vLLM, each harder to setup but opens the door for squeezing more performance out of your hardware. I've also heard okay things about LM Studio and I think Unsloth had their own thing as well: https://unsloth.ai/docs/new/studio >", "context": "", "use_case": "running local LLMs", "switched_from": "", "story": "Which tools do Claude, Codex and Cursor choose? We measured 17k runs to find out"}, {"id": 791902592, "item_id": 49493912, "url": "https://news.ycombinator.com/item?id=49493912", "by": "joshheitzman", "date": "2026-08-29", "stance": "warn", "quote": "I was hoping to see the reasoning_content mess get robustly fixed", "excerpt": "I was hoping to see the reasoning_content mess get robustly fixed, but all we got was this doc change: https://github.com/vllm-project/vllm/pull/50624", "context": "", "use_case": "LLM inference", "switched_from": "", "story": "vLLM v0.28.0"}, {"id": 791896689, "item_id": 49493543, "url": "https://news.ycombinator.com/item?id=49493543", "by": "bugglebeetle", "date": "2026-08-29", "stance": "warn", "quote": "vLLM seems to have a lot more gotchas and kludged together stuff", "excerpt": "I recently switched to using for some large volume inference with small, fine-tuned models and was surprised how nice and painless things were. vLLM seems to have a lot more gotchas and kludged together stuff once you get outside of anything straightforward.", "context": "", "use_case": "LLM inference", "switched_from": "", "story": "vLLM v0.28.0"}, {"id": 791889696, "item_id": 49493106, "url": "https://news.ycombinator.com/item?id=49493106", "by": "kouteiheika", "date": "2026-08-29", "stance": "warn", "quote": "I love vLLM, but damn if it isn't frustratingly buggy.", "excerpt": "I love vLLM, but damn if it isn't frustratingly buggy. I was recently running DeepSeek-V4-Flash on a B300. On v0.26 it was totally broken, and I had to add three out-of-tree patches to fix it. I updated to v0.27 -- no patches necessary now, but the output is now broken as it randomly starts responding with garbage (repeated token loops). On my workstation where I run Gemma-4 on an RTX 6000 the whole process tends to ", "context": "Runs DeepSeek-V4-Flash on a B300 and Gemma-4 on an RTX 6000", "use_case": "serving LLMs on GPUs", "switched_from": "", "story": "vLLM v0.28.0"}, {"id": 790556496, "item_id": 49409781, "url": "https://news.ycombinator.com/item?id=49409781", "by": "TofuLover", "date": "2026-08-23", "stance": "recommend", "quote": "vLLM gang all the way!", "excerpt": "These are great questions, thanks! I'll answer them in turn, in a list because I like lists. * Confidentiality: no logging, third party analytics, or anything like that. More details in our Privacy Poilicy [1]. Our hosting providers will have their own policies, but we're not running a super private service like Proton or similar. Might do some kind of secure tenancy in the future if there's demand. * Price: I think ", "context": "runs an open-model API service", "use_case": "serving open models", "switched_from": "", "story": "I gave Qwen 3.8 27B a reverse-engineering job and it finished in 30 minutes"}, {"id": 790464065, "item_id": 49404004, "url": "https://news.ycombinator.com/item?id=49404004", "by": "JacobJack", "date": "2026-08-22", "stance": "compare", "quote": "I thought the main advantage of VLLM was better concurrency management (better batching).", "excerpt": "> And the comparisons in this post are not going to be running some 2.58-bit-gguf-in-ollama with a couple test prompts. Genuine question : is there something fundamentally wrong with Ollama ? I use Ollama because it is easy to set up and manage (and also because VLLM is not super Windows friendly). I thought the main advantage of VLLM was better concurrency management (better batching). But if the quality of the inte", "context": "", "use_case": "LLM inference serving", "switched_from": "", "story": "Why your local LLM feels dumber than it is"}, {"id": 789605057, "item_id": 49350316, "url": "https://news.ycombinator.com/item?id=49350316", "by": "acd", "date": "2026-08-18", "stance": "neutral", "quote": "Glq runs with vLLM.", "excerpt": "I have written a llm compression quantization library called glq because of the high ram prices. Glq uses qtip Trellis quantization which are efficient at low bpw 2-4 bits. As a gamer and ai developer I want to squeeze more out of the same hardware. Glq runs with vLLM. Open source https://github.com/cnygaard/glq Pypi glq https://pypi.org/project/glq/", "context": "", "use_case": "LLM inference", "switched_from": "", "story": "Memory prices climb 500% in 12 months, up to 10x the lowest ever tracked prices"}, {"id": 789040080, "item_id": 49315005, "url": "https://news.ycombinator.com/item?id=49315005", "by": "anotherCodder", "date": "2026-08-15", "stance": "neutral", "quote": "vllm on 4x 5090 is getting ~20 tok/s with mtp on", "excerpt": "vllm on 4x 5090 is getting ~20 tok/s with mtp on (their own thread on the hf card). i had qwen3.8-27b up the day after release, one rtx pro 6000, 140 tok/s spec, 0.156s first token, full 262k. image and video on the same api. numbers: https://github.com/avifenesh/memra try it: https://inference.tiyuvta.ai/app $0.38 in / $0.20 cache / $2.60 out. openrouter's only host right now is 23 tok/s at $0.45 / $3.20.", "context": "", "use_case": "serving Qwen locally", "switched_from": "", "story": "Qwen 3.8 27B is out: open weights, best local dense model yet"}, {"id": 788973504, "item_id": 49310844, "url": "https://news.ycombinator.com/item?id=49310844", "by": "sgt", "date": "2026-08-15", "stance": "warn", "quote": "I already tried using vLLM but it ran out of GPU memory.", "excerpt": "Amazing. I'll give it a shot on my 5090. I already tried using vLLM but it ran out of GPU memory. I guess it's likely Llama.cpp will work.", "context": "RTX 5090 owner", "use_case": "running Qwen locally", "switched_from": "", "story": "Qwen 3.8 27B is out: open weights, best local dense model yet"}, {"id": 788312928, "item_id": 49269558, "url": "https://news.ycombinator.com/item?id=49269558", "by": "embedding-shape", "date": "2026-08-12", "stance": "neutral", "quote": "I mostly end up using vLLM these days", "excerpt": "> At this point the options are llama-server or vLLM Which last time I checked, both use different formats of the weights, the former GGUF while the latter .safetensors. I mostly end up using vLLM these days and I'm a bit more performance sensitive than what I used to be. Just a shame it's a hassle to share the weights between them with conversion and what not, either batched or on-startup.", "context": "performance sensitive user", "use_case": "running local models", "switched_from": "", "story": "llama.cpp"}, {"id": 788183124, "item_id": 49261445, "url": "https://news.ycombinator.com/item?id=49261445", "by": "yummyelephant8", "date": "2026-08-11", "stance": "neutral", "quote": "hooked it up to vLLM on Modal", "excerpt": "Like you, I use coding agents like Claude Code, Codex, and Cursor every day. But the other day, I was curious about how Cursor trained their original tab-completion model. So I started wondering: could you post-train an open-source LLM to autocomplete ComfyUI workflows given your current workflow, similar to how Composer autocompletes code? More specifically: Could I post-train an LLM to predict up to four nodes in t", "context": "", "use_case": "model inference serving", "switched_from": "", "story": null}, {"id": 786356290, "item_id": 49147268, "url": "https://news.ycombinator.com/item?id=49147268", "by": "redrove", "date": "2026-08-02", "stance": "neutral", "quote": "I\u2019ve been running a custom VLLM image with b12x", "excerpt": "The FP8 version from DeepSeek themselves [0], around 1800 tps prefill and 45 tokens per second decode. I\u2019ve been running a custom VLLM image with b12x as well as nvfp4_ds_mla. I would say it\u2019s quite fantastic in day to day, I use it mostly in Hermes and sometimes for coding. I have qwen 3.6 27b in an rtx 6000 pro as well so I use that as a workhorse in pi with DS as a reviewer/planner. [0] https://huggingface.co/deep", "context": "", "use_case": "model serving", "switched_from": "", "story": "Show HN: NixOS-DGX-Spark \u2013 Nix and NixOS on the DGX Spark"}, {"id": 785962608, "item_id": 49122663, "url": "https://news.ycombinator.com/item?id=49122663", "by": "rurban", "date": "2026-07-31", "stance": "recommend", "quote": "For LLM inference in production we use vLLM (with our own model).", "excerpt": "For LLM inference in production we use vLLM (with our own model).", "context": "", "use_case": "LLM inference in production", "switched_from": "", "story": "Ask HN: What are you using for LLM inference in production?"}, {"id": 784353520, "item_id": 49022095, "url": "https://news.ycombinator.com/item?id=49022095", "by": "scottcha", "date": "2026-07-23", "stance": "recommend", "quote": "We use vllm as it generally has the best ecosystem support.", "excerpt": "We use vllm as it generally has the best ecosystem support. Parameters are largely dependent on what type of requests you are serving (concurrency, input/output ratios, cached hit patterns). We've never had a limitation at the tokenizer step. Limitations at peak tend to manifest more on slower time in vllm doing prefill or decode though we actively try and minimize this.", "context": "", "use_case": "LLM serving", "switched_from": "", "story": "GigaToken: ~1000x faster Language model tokenization"}, {"id": 783466787, "item_id": 48966674, "url": "https://news.ycombinator.com/item?id=48966674", "by": "boomskats", "date": "2026-07-19", "stance": "neutral", "quote": "w/ the big cohere-transcribe running on a local vLLM instance", "excerpt": "I have a personal fork of hyprvoice[0] which I use almost everywhere now (w/ the big cohere-transcribe running on a local vLLM instance). It does a similar thing, but that's not why I'm mentioning it; I think it's worth looking at because it's a clean reference for the few elegant ways you can implement text injection in modern Linux (wayland). It supports ydotool[1], wtype[2] and \"clipboard fallback with clipboard r", "context": "", "use_case": "local transcription serving", "switched_from": "", "story": "Transcribe.cpp"}, {"id": 782605186, "item_id": 48912824, "url": "https://news.ycombinator.com/item?id=48912824", "by": "verdverm", "date": "2026-07-14", "stance": "neutral", "quote": "Preliminary analysis via lm-evaluation-harness + vllm", "excerpt": "Preliminary analysis via lm-evaluation-harness + vllm model | disk | wikitext | gsm8k (match/error) baseline | 55G | 8.00 | 0.50/0.09 nvfp4-gptq | 27G | 8.25 | 0.47/0.9 nvfp4a16-gptq | 27G | 8.11 | 0.53/0.9 bonsai-4bit | 19G | 16.75 | 0/0 Looks like they quant'd too hard at 4 bits, can't imagine the ternary being any good based on this. Code if you'd like to reproduce or try other test sets: https://github.com/verdve", "context": "", "use_case": "model inference", "switched_from": "", "story": "Bonsai 27B (1-bit LLM): The First 27B-Class Model to Run on a Phone"}, {"id": 782174896, "item_id": 48885931, "url": "https://news.ycombinator.com/item?id=48885931", "by": "blinded", "date": "2026-07-12", "stance": "neutral", "quote": "you need to run vllm and a model", "excerpt": "Hey sure. It depends, but usually spin up an h100 on lambda.ai or coreweave. They have capacity and their UIs/APIs are nice. I spin it up for an hour or two, believe it was 6~ dollars an hour. Once the gpu instance is up, you need to run vllm and a model, ie https://docs.lambda.ai/education/large-language-models/deploying-nemotron-3-nano/. Then you can connect your pi.dev, openwebui, etc etc to vllm and interact with", "context": "", "use_case": "serving private LLM on rented GPU", "switched_from": "", "story": "Ask HN: How do you use LLMs for private discussions?"}, {"id": 782166928, "item_id": 48885433, "url": "https://news.ycombinator.com/item?id=48885433", "by": "blinded", "date": "2026-07-12", "stance": "recommend", "quote": "I spin up a gpu instance in a cloud, run my model via vllm, connect to it via an ssh tunnel. done.", "excerpt": "I spin up a gpu instance in a cloud, run my model via vllm, connect to it via an ssh tunnel. done.", "context": "", "use_case": "private LLM serving", "switched_from": "", "story": "Ask HN: How do you use LLMs for private discussions?"}, {"id": 779940113, "item_id": 48746257, "url": "https://news.ycombinator.com/item?id=48746257", "by": "coder543", "date": "2026-07-01", "stance": "warn", "quote": "vLLM is very painful, even with the Spark community docker image.", "excerpt": "It\u2019s not FUD. It is my actual, lived experience. FUD is false, which this is not. I use both vLLM and llama-server. vLLM is very painful, even with the Spark community docker image. It is slow to start, it does not support 3-bit dynamic quants well, and it takes a lot of tweaking to get it to run well for each model I want to try out, which is made worse by the slow starts. I\u2019m glad you\u2019ve had a better experience? I ", "context": "runs local models on a Spark", "use_case": "local LLM inference", "switched_from": "", "story": "Qwen 3.6 27B is the sweet spot for local development"}, {"id": 777904161, "item_id": 48619010, "url": "https://news.ycombinator.com/item?id=48619010", "by": "pet_the_bird", "date": "2026-06-21", "stance": "neutral", "quote": "I have tried various self-hosted models using both vllm and llama.cpp.", "excerpt": "Highy anecdotal: I have tried various self-hosted models using both vllm and llama.cpp. I am in a situation where I have access to large amount of memory (~320 GB). While experimenting with quantization I found that there is a non-trivial tradeoff between quality and memory footprint. Overall my experience follows the reported pattern of \"2-bit is mwah, 4-bit half decent and 6-bit required for programming. Still, alt", "context": "has access to ~320 GB memory", "use_case": "self-hosting models", "switched_from": "", "story": "Two Qwen3 models on one DGX Spark: the residency math"}, {"id": 777398657, "item_id": 48587416, "url": "https://news.ycombinator.com/item?id=48587416", "by": "alexellisuk", "date": "2026-06-18", "stance": "compare", "quote": "We did run vLLM on the 3090s \u2014 measured ~3 tok/s slower on generation for our single-to-few-user pattern", "excerpt": "We did run vLLM on the 3090s \u2014 measured ~3 tok/s slower on generation for our single-to-few-user pattern, plus less flexibility on quant and slower startup (actual minutes vs single digit seconds). We may do more with it again in the future - there isn't unlimited time for us to tinker, I'm sharing our journey (so far) and reasoning. It's the right call for concurrent batched serving (barrkel's point downthread is sp", "context": "", "use_case": "LLM serving", "switched_from": "", "story": "Local Qwen isn't a worse Opus, it's a different tool"}, {"id": 776726594, "item_id": 48545412, "url": "https://news.ycombinator.com/item?id=48545412", "by": "jodoherty", "date": "2026-06-15", "stance": "compare", "quote": "I switch between vLLM using nvidia/Gemma-4-31B-IT-NVFP4 and llama.cpp", "excerpt": "I use pi with an RTX Pro 6000 Blackwell to run Gemma 4 31b to do all my agentic coding. I find it useful. This side project highlights a similar approach to how I scope and tackle projects at work now: https://git.theodohertyfamily.com/wg-wrap.git/tree/README.md https://git.theodohertyfamily.com/wg-wrap.git/tree/CASE_STUDY.md You have to apply a lot of careful architecture and TDD to your approach. Eliminate technica", "context": "", "use_case": "serving local model", "switched_from": "", "story": "Ask HN: Has anyone replaced Claude/GPT with a local model for daily coding?"}, {"id": 775778243, "item_id": 48486140, "url": "https://news.ycombinator.com/item?id=48486140", "by": "stevefan1999", "date": "2026-06-10", "stance": "recommend", "quote": "using just the ollama, llama.cpp containers and especially vllm operator", "excerpt": "> Ever hear of a large K8s migration going well ...well, AI infra stacks are more or less k8s/cloud native, especially on the inference side. Nvidia GPU operator plus KubeFlow makes deploying models easy, and I did that manually as well using just the ollama, llama.cpp containers and especially vllm operator. I'm not sure about what's the measuring notation of \"large\" is, but OpenAI had a blog post about how to manag", "context": "", "use_case": "model serving", "switched_from": "", "story": "Claude Desktop spins up a VM without no way of stopping it"}, {"id": 774215824, "item_id": 48388489, "url": "https://news.ycombinator.com/item?id=48388489", "by": "verdverm", "date": "2026-06-03", "stance": "recommend", "quote": "I switched from llama.cpp to vLLM because of prompt cache bugs in qwen/gemma models", "excerpt": "I switched from llama.cpp to vLLM because of prompt cache bugs in qwen/gemma models This is a good starting issue with a bunch of linked/related https://github.com/ggml-org/llama.cpp/issues/22746", "context": "", "use_case": "local model inference", "switched_from": "llama.cpp", "story": "Gemma 4 12B: A unified, encoder-free multimodal model"}, {"id": 773231073, "item_id": 48326942, "url": "https://news.ycombinator.com/item?id=48326942", "by": "verdverm", "date": "2026-05-29", "stance": "recommend", "quote": "OpenCode + vllm, model will depend on your hardware", "excerpt": "OpenCode + vllm, model will depend on your hardware, but OpenCode also has a killer $10/m plan with quotas for some top tier open weight models. I'm using qwen3.6 on a DGX spark, llama-cpp has prompt cache bugs for qwen/gemma models (among more being reported). Using my OpenCode-go sub when I want a bigger / more capable model", "context": "runs models on a DGX Spark", "use_case": "local model serving", "switched_from": "", "story": "Ask HN: If I cancel Codex today whats the next best local inference agent?"}, {"id": 768907056, "item_id": 48056691, "url": "https://news.ycombinator.com/item?id=48056691", "by": "cookiengineer", "date": "2026-05-07", "stance": "warn", "quote": "You still can't build the vllm container with updated dependencies since llmlite got pwned.", "excerpt": "Fun fact: You still can't build the vllm container with updated dependencies since llmlite got pwned. Either due to regression bugs, or due to impossible transient dependencies in the dependency tree that are not resolvable. There is just too much slopcode down the line, and too many dependencies relying on pinned outdated (and unpublished) dependencies. I switched to llama.cpp because of that. To me it feels more an", "context": "", "use_case": "LLM inference", "switched_from": "", "story": "Maybe you shouldn't install new software for a bit"}, {"id": 766015360, "item_id": 47875960, "url": "https://news.ycombinator.com/item?id=47875960", "by": "AlexC04", "date": "2026-04-23", "stance": "recommend", "quote": "vLLM apparently already has an implementation of turboquant available", "excerpt": "One other thing you might want to check out for running locally. (I have not independently verified yet, it's on the TODO list though) https://docs.vllm.ai/en/latest/api/vllm/model_executor/layers/quantization/turboquant/ vLLM apparently already has an implementation of turboquant available - which is said to losslessly reduce the memory footprint required by 6x and improve inference speed by 8x. From what I understa", "context": "developer-tools project, two young kids", "use_case": "local LLM inference", "switched_from": "", "story": "Qwen3.6-27B: Flagship-Level Coding in a 27B Dense Model"}, {"id": 755129489, "item_id": 47195593, "url": "https://news.ycombinator.com/item?id=47195593", "by": "deepsquirrelnet", "date": "2026-02-28", "stance": "neutral", "quote": "I only wish gguf format had better vllm support.", "excerpt": "I love the work unsloth is doing. I only wish gguf format had better vllm support. It\u2019s sometimes hard to find trustworthy quants that work well with vllm.", "context": "", "use_case": "model serving", "switched_from": "", "story": "Unsloth Dynamic 2.0 GGUFs"}, {"id": 750936451, "item_id": 46933528, "url": "https://news.ycombinator.com/item?id=46933528", "by": "graphpilled", "date": "2026-02-08", "stance": "neutral", "quote": "The whole thing is served on a single 4090 GPU using vLLM.", "excerpt": "Hi HN, I'm a computer systems engineering student in Mexico who switched from film school. I built CineGraphs because my filmmaker friends and I kept hitting the same wall\u2014we'd have a vague idea for a film but no structured way to explore where it could go. Every AI writing tool we tried output generic, formulaic slop. I didn't want to build another ChatGPT wrapper, so I went a different route. The idea is simple: yo", "context": "", "use_case": "serving model", "switched_from": "", "story": "Show HN: Fine-tuned Qwen2.5-7B on 100 films for probabilistic story graphs"}, {"id": 749072225, "item_id": 46817014, "url": "https://news.ycombinator.com/item?id=46817014", "by": "Imustaskforhelp", "date": "2026-01-29", "stance": "compare", "quote": "Comparable inference speeds to vLLM", "excerpt": "> My dream would be something like vLLM, but without all the Python mess, packaged as a single binary that has both HTTP server + desktop GUI, and can browse/download models. Llama.cpp is like 70% there, but large performance difference between llama.cpp and vLLM for the models I use. To be honest, I was seeing your comment multiple times and after 6 hours, It suddenly clicked about something new. I had seen this pro", "context": "", "use_case": "LLM inference", "switched_from": "", "story": "LM Studio 0.4.0"}, {"id": 746904496, "item_id": 46681531, "url": "https://news.ycombinator.com/item?id=46681531", "by": "jakeasmith", "date": "2026-01-19", "stance": "neutral", "quote": "I wanted to experiment with running something a little more \"low level\".", "excerpt": "I've been running Ollama in my home lab for awhile now but I wanted to experiment with running something a little more \"low level\". I saw a ton of posts about Llama.cpp, which looked interesting, but there wasn't a lot specifically about running vLLM on Proxmox, so I thought I'd give it a try. Setting up vLLM in an LXC wasn\u2019t necessarily difficult but, even after doing it several times, it was still tedious. These ar", "context": "home lab user", "use_case": "LLM serving in Proxmox LXC", "switched_from": "Ollama", "story": "Running a vLLM LXC on Proxmox 9 with Nvidia GPU Passthrough"}, {"id": 736557456, "item_id": 46034841, "url": "https://news.ycombinator.com/item?id=46034841", "by": "Yarden_Bruch_El", "date": "2025-11-24", "stance": "neutral", "quote": "We built a custom inference layer on top of vLLM", "excerpt": "Hi HN, We built a platform for orchestrating multi-agent debates (e.g., \"Security\" vs. \"Refactoring\" experts). The Challenge: Standard sequential agent chains (A -> B -> C) are too slow for real-time chat. The Fix (vLLM): We built a custom inference layer on top of vLLM to solve the bottleneck: Parallelism: We use continuous batching to generate multiple agent responses simultaneously rather than waiting for sequenti", "context": "", "use_case": "parallel multi-agent inference", "switched_from": "", "story": "BYO \u2013 A multi-agent runtime optimized for parallel inference"}, {"id": 732496465, "item_id": 45781029, "url": "https://news.ycombinator.com/item?id=45781029", "by": "embedding-shape", "date": "2025-11-01", "stance": "recommend", "quote": "Need batching + highest total throughoutput? vLLM, complicated to deploy and install though", "excerpt": "Not parent, but frequent user of GPT-OSS, tried all different ways of running it. Choice goes something like this: - Need batching + highest total throughoutput? vLLM, complicated to deploy and install though, need special versions for top performance with GPT-OSS - Easiest to manage + fast enough: llama.cpp, easier to deploy as well (just a binary) and super fast, getting ~260 tok/s on a RTX Pro 6000 for the 20B ver", "context": "", "use_case": "high-throughput batched serving", "switched_from": "", "story": "Ask HN: Who uses open LLMs and coding assistants locally? Share setup and laptop"}]}