{"slug": "llama-server", "name": "llama-server", "category": "ai", "category_name": "AI models and assistants", "description": "", "recommenders": 3, "commenters": 8, "warners": 1, "score": 2.927, "rising": 1.67, "first": "2026-04-27", "latest": "2026-10-02", "why": "llama-server also comes with a truly excellent built-in web chat interface these days", "why_is_quote": true, "has_verdict": false, "sentiment": {"recommend": 3, "neutral": 3, "compare": 1, "warn": 1}, "spark": [0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0], "listed": true, "loved": [], "caveat": "", "short_quote": "llama-server also comes with a truly excellent built-in web chat interface these days", "icon": true, "domain": null, "github_repo": null, "official_url": "https://github.com/ggml-org/llama.cpp", "pricing": "", "aliases": ["llama-server"], "verdict": null, "praise": [{"label": "", "summary": "", "quotes": [{"id": 796066880, "item_id": 49754180, "url": "https://news.ycombinator.com/item?id=49754180", "by": "jakswa", "date": "2026-09-18", "stance": "recommend", "quote": "I love their web UI so much", "excerpt": "I love their web UI so much that I had AI slop all over it in a fit of fanboy-ism https://inkcap.click", "context": "", "use_case": "local LLM web UI", "switched_from": "", "story": "Bonsai 2 27B: Near-Lossless Compression in a 9x Smaller Footprint"}, {"id": 791703073, "item_id": 49481442, "url": "https://news.ycombinator.com/item?id=49481442", "by": "wolvoleo", "date": "2026-08-28", "stance": "recommend", "quote": "I use it with llama-server and openwebui.", "excerpt": "Similar here, except I used Opus 4.8. I don't mind using cloud models to make code as long as I don't put actual private data into the model. And I can't run qwen 27b reliably at the moment, I need a better gpu. It was pretty easy mode like this tbh, though you do need to know what you're doing. I use it with llama-server and openwebui. I guess you can get it a lot more click to go with something like LM Studio thoug", "context": "", "use_case": "serving local models", "switched_from": "", "story": "Run Qwen3.8 27B locally: real numbers from my Mac Studio"}, {"id": 779940112, "item_id": 48746257, "url": "https://news.ycombinator.com/item?id=48746257", "by": "coder543", "date": "2026-07-01", "stance": "recommend", "quote": "llama-server also comes with a truly excellent built-in web chat interface these days", "excerpt": "It\u2019s not FUD. It is my actual, lived experience. FUD is false, which this is not. I use both vLLM and llama-server. vLLM is very painful, even with the Spark community docker image. It is slow to start, it does not support 3-bit dynamic quants well, and it takes a lot of tweaking to get it to run well for each model I want to try out, which is made worse by the slow starts. I\u2019m glad you\u2019ve had a better experience? I ", "context": "runs local models on a Spark", "use_case": "local LLM inference", "switched_from": "vLLM", "story": "Qwen 3.6 27B is the sweet spot for local development"}]}], "caveats": [{"label": "", "summary": "", "quotes": [{"id": 793811552, "item_id": 49613222, "url": "https://news.ycombinator.com/item?id=49613222", "by": "bellowsgulch", "date": "2026-09-08", "stance": "warn", "quote": "llama-server the last time I used it for inference with this model wasn\u2019t able to produce work fast enough to reach those numbers.", "excerpt": "Thanks, man! I\u2019ll go use that now that I know. llama-server the last time I used it for inference with this model wasn\u2019t able to produce work fast enough to reach those numbers.", "context": "", "use_case": "local model inference", "switched_from": "", "story": "Benchmarking Qwen3.8 27B quantizations: 4-bit holds up, 1-bit collapses"}]}], "who": [{"by": "coder543", "context": "runs local models on a Spark", "url": "https://news.ycombinator.com/item?id=48746257", "date": "2026-07-01"}], "tags": [], "use_cases": ["serving local models", "running local models", "local llm web ui", "local model inference", "running kimi k3 locally", "local llm inference"], "compared": [{"slug": "vllm", "name": "vLLM", "co": 2, "switched_from": 1, "switched_to": 0}, {"slug": "mtplx", "name": "MTPLX", "co": 1, "switched_from": 0, "switched_to": 1}, {"slug": "ollama", "name": "Ollama", "co": 1, "switched_from": 1, "switched_to": 0}, {"slug": "open-webui", "name": "Open WebUI", "co": 2, "switched_from": 0, "switched_to": 0}], "makers": [], "threads": [{"url": "https://news.ycombinator.com/item?id=49926069", "title": "Pi 1.0", "mentions": 1}, {"url": "https://news.ycombinator.com/item?id=49746618", "title": "Bonsai 2 27B: Near-Lossless Compression in a 9x Smaller Footprint", "mentions": 1}, {"url": "https://news.ycombinator.com/item?id=49611128", "title": "Benchmarking Qwen3.8 27B quantizations: 4-bit holds up, 1-bit collapses", "mentions": 1}, {"url": "https://news.ycombinator.com/item?id=49479951", "title": "Run Qwen3.8 27B locally: real numbers from my Mac Studio", "mentions": 1}, {"url": "https://news.ycombinator.com/item?id=49267928", "title": "llama.cpp", "mentions": 1}, {"url": "https://news.ycombinator.com/item?id=49123386", "title": "Run Kimi K3 using 29 GB of RAM at 0.50 tok/s", "mentions": 1}], "sentiment_by_month": {"2025-11": {}, "2025-12": {}, "2026-01": {}, "2026-02": {}, "2026-03": {}, "2026-04": {"neutral": 1}, "2026-05": {}, "2026-06": {}, "2026-07": {"recommend": 1}, "2026-08": {"neutral": 1, "compare": 1, "recommend": 1}, "2026-09": {"warn": 1, "recommend": 1}, "2026-10": {"neutral": 1}}, "quotes": [{"id": 798939810, "item_id": 49933738, "url": "https://news.ycombinator.com/item?id=49933738", "by": "tingletech", "date": "2026-10-02", "stance": "neutral", "quote": "I use it with a local Qwen3.8 27B on llama-server", "excerpt": "I use it with a local Qwen3.8 27B on llama-server, but I don't know what it means for it exceed any window. you can use /slots on the llama-server if you want to get more up-to-date details on session token use. I always run models in their default context, which is 262144 for Qwen3.8 27B. I've run sessions in hax where it hit that cap multiple times and compressed the context down to 15% and continued with no proble", "context": "", "use_case": "serving local models", "switched_from": "", "story": "Pi 1.0"}, {"id": 796066880, "item_id": 49754180, "url": "https://news.ycombinator.com/item?id=49754180", "by": "jakswa", "date": "2026-09-18", "stance": "recommend", "quote": "I love their web UI so much", "excerpt": "I love their web UI so much that I had AI slop all over it in a fit of fanboy-ism https://inkcap.click", "context": "", "use_case": "local LLM web UI", "switched_from": "", "story": "Bonsai 2 27B: Near-Lossless Compression in a 9x Smaller Footprint"}, {"id": 793811552, "item_id": 49613222, "url": "https://news.ycombinator.com/item?id=49613222", "by": "bellowsgulch", "date": "2026-09-08", "stance": "warn", "quote": "llama-server the last time I used it for inference with this model wasn\u2019t able to produce work fast enough to reach those numbers.", "excerpt": "Thanks, man! I\u2019ll go use that now that I know. llama-server the last time I used it for inference with this model wasn\u2019t able to produce work fast enough to reach those numbers.", "context": "", "use_case": "local model inference", "switched_from": "", "story": "Benchmarking Qwen3.8 27B quantizations: 4-bit holds up, 1-bit collapses"}, {"id": 791703073, "item_id": 49481442, "url": "https://news.ycombinator.com/item?id=49481442", "by": "wolvoleo", "date": "2026-08-28", "stance": "recommend", "quote": "I use it with llama-server and openwebui.", "excerpt": "Similar here, except I used Opus 4.8. I don't mind using cloud models to make code as long as I don't put actual private data into the model. And I can't run qwen 27b reliably at the moment, I need a better gpu. It was pretty easy mode like this tbh, though you do need to know what you're doing. I use it with llama-server and openwebui. I guess you can get it a lot more click to go with something like LM Studio thoug", "context": "", "use_case": "serving local models", "switched_from": "", "story": "Run Qwen3.8 27B locally: real numbers from my Mac Studio"}, {"id": 788312929, "item_id": 49269558, "url": "https://news.ycombinator.com/item?id=49269558", "by": "embedding-shape", "date": "2026-08-12", "stance": "compare", "quote": "Which last time I checked, both use different formats of the weights, the former GGUF while the latter .safetensors.", "excerpt": "> At this point the options are llama-server or vLLM Which last time I checked, both use different formats of the weights, the former GGUF while the latter .safetensors. I mostly end up using vLLM these days and I'm a bit more performance sensitive than what I used to be. Just a shame it's a hassle to share the weights between them with conversion and what not, either batched or on-startup.", "context": "", "use_case": "running local models", "switched_from": "", "story": "llama.cpp"}, {"id": 786111008, "item_id": 49131938, "url": "https://news.ycombinator.com/item?id=49131938", "by": "walrus01", "date": "2026-08-01", "stance": "neutral", "quote": "I plan to give it a try in a day or two with llama-server from the main branch", "excerpt": "I plan to give it a try in a day or two with llama-server from the main branch compiled today, when my Q8 GGUF download of K3 finishes, on a system with 256GB (should be more than ample for context and KV cache and a moderate chunk of the whole 1.6TB). If it works it's going to be sloooooooow as hell, but it'll be an interesting data point to see just how slow.", "context": "", "use_case": "running Kimi K3 locally", "switched_from": "", "story": "Run Kimi K3 using 29 GB of RAM at 0.50 tok/s"}, {"id": 779940112, "item_id": 48746257, "url": "https://news.ycombinator.com/item?id=48746257", "by": "coder543", "date": "2026-07-01", "stance": "recommend", "quote": "llama-server also comes with a truly excellent built-in web chat interface these days", "excerpt": "It\u2019s not FUD. It is my actual, lived experience. FUD is false, which this is not. I use both vLLM and llama-server. vLLM is very painful, even with the Spark community docker image. It is slow to start, it does not support 3-bit dynamic quants well, and it takes a lot of tweaking to get it to run well for each model I want to try out, which is made worse by the slow starts. I\u2019m glad you\u2019ve had a better experience? I ", "context": "runs local models on a Spark", "use_case": "local LLM inference", "switched_from": "vLLM", "story": "Qwen 3.6 27B is the sweet spot for local development"}, {"id": 766873601, "item_id": 47929600, "url": "https://news.ycombinator.com/item?id=47929600", "by": "anandkrshnn", "date": "2026-04-27", "stance": "neutral", "quote": "Going to try llama-server today.", "excerpt": "Thanks for laying out the timeline so clearly. I\u2019ve been using Ollama for months because it \u2018just worked,\u2019 but I had no idea about the missing attribution or the hashed blob storage lock-in. The fact that I can\u2019t easily point another tool at the same GGUF file is a dealbreaker for me. Going to try llama-server today.", "context": "", "use_case": "running local models", "switched_from": "Ollama", "story": "Stop Using Ollama"}]}