{"slug": "llama-cpp", "name": "llama.cpp", "category": "ai", "category_name": "AI models and assistants", "description": "Open-source C/C++ engine for running LLM inference locally on CPUs and GPUs.", "recommenders": 69, "commenters": 135, "warners": 4, "score": 73.203, "rising": 16.58, "first": "2025-11-01", "latest": "2026-10-20", "why": "Recommended over Ollama as a faster, more tunable way to run local models, with a built-in server, web UI and broad GPU backend support.", "why_is_quote": false, "has_verdict": true, "sentiment": {"recommend": 68, "neutral": 69, "compare": 6, "warn": 5}, "spark": [2, 2, 0, 2, 1, 4, 4, 16, 7, 17, 11, 2], "listed": true, "loved": ["Ollama alternative", "Performance tuning"], "caveat": "GPU setup", "short_quote": "I highly recommend using llama.cpp for running AI model locally vs using other inference framework", "icon": true, "domain": null, "github_repo": "github.com/ggml-org/llama.cpp", "official_url": "https://github.com/ggml-org/llama.cpp", "pricing": "open source", "aliases": ["llama.cpp", "Llama.cpp"], "verdict": [{"text": "Recommended over Ollama as a faster, more tunable way to run local models", "cites": [{"id": 793020256, "item_id": 49563766, "url": "https://news.ycombinator.com/item?id=49563766", "by": "eloisant", "date": "2026-09-04", "stance": "recommend", "quote": "I recommend llama.cpp over ollama.", "excerpt": "I recommend llama.cpp over ollama. Anyway, when you see a model on huggingface you get the exact commands to copy paste to run it locally.", "context": "", "use_case": "running local models", "switched_from": "", "story": "Which tools do Claude, Codex and Cursor choose? We measured 17k runs to find out"}, {"id": 776749920, "item_id": 48546870, "url": "https://news.ycombinator.com/item?id=48546870", "by": "jboss10", "date": "2026-06-15", "stance": "recommend", "quote": "llama.cpp It's faster and more open source.", "excerpt": "llama.cpp It's faster and more open source. Ollama has some mixed history. I use llama-swap to emulate the Ollama experience.", "context": "", "use_case": "local LLM inference", "switched_from": "", "story": "Ask HN: Has anyone replaced Claude/GPT with a local model for daily coding?"}, {"id": 790471296, "item_id": 49404456, "url": "https://news.ycombinator.com/item?id=49404456", "by": "zepearl", "date": "2026-08-22", "stance": "recommend", "quote": "I now absolutely love all the knobs that \"llama.cpp\" has to tune your hardware setup & your workload", "excerpt": "I know nothing about \"vLLM\" -> anything better compared to \"llama.cpp\"? I started with \"Ollama\" (precompiled version) and it worked and was good enough to understand the very basics. Then I downloaded the sourcecode of \"llama.cpp\", compiled it with specific compilation options for my GPUs (CUDA/nVidia using proprietary module on Gentoo Linux) & CPU (AMD), and the same model ran twice as fast -> since then I stuck wit", "context": "Gentoo Linux user with NVIDIA GPUs and AMD CPU", "use_case": "running local LLMs", "switched_from": "Ollama", "story": "Why your local LLM feels dumber than it is"}, {"id": 783500929, "item_id": 48968808, "url": "https://news.ycombinator.com/item?id=48968808", "by": "ollamer", "date": "2026-07-19", "stance": "recommend", "quote": "On llama.cpp I can get models like Qwen 35BA3B running partially on gpu/cpu with 40t/s on a laptop thanks to --n-cpu-moe", "excerpt": "https://github.com/ollama/ollama/issues/11772 A year and still no implementation for such a basic need as offloading MoE layers onto the CPU selectively. On llama.cpp I can get models like Qwen 35BA3B running partially on gpu/cpu with 40t/s on a laptop thanks to --n-cpu-moe but on this VC funded joke it would be simply unusable. I can't quite understand how you make a wrapper so much worse than the code you're rippin", "context": "", "use_case": "partial CPU/GPU MoE inference", "switched_from": "", "story": "Ollama: All Aboard Open Models"}]}, {"text": ", with a built-in server, web UI and broad GPU backend support.", "cites": [{"id": 794057472, "item_id": 49628592, "url": "https://news.ycombinator.com/item?id=49628592", "by": "tarruda", "date": "2026-09-09", "stance": "recommend", "quote": "llama.cpp is simply better with its awesome built-in webui, router and server APIs", "excerpt": "I tried dwarfstar when llama.cpp DSV4 support was still very weak, and while it worked, I didn't see anything that would make me want to stick with it vs llama.cpp. llama.cpp is simply better with its awesome built-in webui, router and server APIs and certainly support much more models and quantizations than dwarfstar. Since then, I started maintaining my own vibe coded dsv4 branch with metal optimizations, so I actu", "context": "", "use_case": "running DeepSeek V4 locally", "switched_from": "dwarfstar", "story": "DeepSeek launching v4.1 flash cheaper and more capable than v4 pro"}, {"id": 782202704, "item_id": 48887669, "url": "https://news.ycombinator.com/item?id=48887669", "by": "MyMemoryfails", "date": "2026-07-12", "stance": "recommend", "quote": "Running Local LLM isn't really a lot effort anymore, since its one command away", "excerpt": "Running Local LLM isn't really a lot effort anymore, since its one command away to get the familar interface of chat AI. This setup requires 24GB ish VRAM and 32GB ram, so if you have capable gaming PC which can do AAA games, you can run it. - llama-server -hf ggml-org/gemma-4-26b-a4b-it-GGUF:Q4_K_M After that simply open browser and enter: http://localhost:8080 What this does: This will download Gemma4 AI with 26B p", "context": "", "use_case": "running local LLMs", "switched_from": "", "story": "Ask HN: How do you use LLMs for private discussions?"}, {"id": 776719776, "item_id": 48544986, "url": "https://news.ycombinator.com/item?id=48544986", "by": "porkloin", "date": "2026-06-15", "stance": "recommend", "quote": "llama.cpp vulkan (better performance than ROCM, though I'm keeping an eye on it in the future)", "excerpt": "I have good results with this setup: Hardware: - GPU: AMD 7900xtx, 24gb vram - CPU: AMD 5950x, AM4 - RAM: 64gb DDR4 3600 Software: - OS: Bazzite (atomic fedora - this machine is running Steam \"big picture\" mode on my TV when not in use for LLM tasks) - Virtualization: Podman Quadlets, which allows me to run container images as managed systemd units - Network: tailscale - Inference: llama.cpp vulkan (better performanc", "context": "", "use_case": "local LLM inference", "switched_from": "", "story": "Ask HN: Has anyone replaced Claude/GPT with a local model for daily coding?"}, {"id": 788644961, "item_id": 49290310, "url": "https://news.ycombinator.com/item?id=49290310", "by": "SwellJoe", "date": "2026-08-13", "stance": "recommend", "quote": "they currently work great with llama.cpp and current ROCm 7.14", "excerpt": "\"used to\" in your sentence already answers your curiosity. ROCm runs most things just fine without any ceremony or difficulty. I have a couple of the same cards as covered in the article, bought before they got expensive (I'd recommend current Radeon AI Pro 9700 cards over the V620 now, though, as they have increased in price since I bought mine), and they're at the long end of the supported chart for ROCm...nearly E", "context": "", "use_case": "local inference", "switched_from": "", "story": "We Have AI at Home Chapter 1: A Box of Scraps"}]}, {"text": " The most common caveat is GPU setup friction", "cites": [{"id": 776006098, "item_id": 48500381, "url": "https://news.ycombinator.com/item?id=48500381", "by": "dlcarrier", "date": "2026-06-11", "stance": "warn", "quote": "I can't get Llama.cpp to use the GPU", "excerpt": "I have an Intel A770, because at $200 USD it was the second cheapest way to get 16 of dedicated VRAM. (The cheapest is a $100 Radeon Instinct Mi25, but it's a bit old and slow.) I battled with Python packages a lot to get a1111 working, gave up when it broke on an update and switched to ComfyUI which has worked great. I can't get Llama.cpp to use the GPU, but I'm currently installing some of Intel's proprietary OpenV", "context": "", "use_case": "LLM on Intel GPU", "switched_from": "", "story": "Ask HN: What computer are you using for AI coding tools?"}, {"id": 784331137, "item_id": 49020696, "url": "https://news.ycombinator.com/item?id=49020696", "by": "gbuk2013", "date": "2026-07-23", "stance": "recommend", "quote": "there was some fiddling required to get it to use GPU (I use llama.cpp)", "excerpt": "I run Qwen3-Coder-Next-UD-Q4_K_XL and other than the initial wait to initialise context (which takes less than 2 minutes) subsequent prompts return in less than a minute, usually less than 30s. If your performance is significantly slower then you are probably doing it in CPU - there was some fiddling required to get it to use GPU (I use llama.cpp)", "context": "", "use_case": "local LLM inference", "switched_from": "", "story": "New Framework Desktop Option with AMD Ryzen AI Max+ Pro 495 and 192GB Memory"}, {"id": 793858096, "item_id": 49616131, "url": "https://news.ycombinator.com/item?id=49616131", "by": "civvv", "date": "2026-09-08", "stance": "recommend", "quote": "so I switched to the Windows Vulkan build of llama.cpp instead, which worked reliably.", "excerpt": "I got Qwen3.8-27B running on my RX 9070 XT by setting up AMD\u2019s ROCm 7.14/TheRock runtime for gfx1201 and adding the required HIP/LLVM paths to Windows. The official llama.cpp ROCm build detected the card correctly, but kept crashing with 0xC0000005 while loading the model, so I switched to the Windows Vulkan build of llama.cpp instead, which worked reliably. I\u2019m running Qwen3.8-27B-Unleashed UD-Q3_K_XL, which is a ~1", "context": "RX 9070 XT user on Windows", "use_case": "running Qwen locally on AMD GPU", "switched_from": "llama.cpp ROCm build", "story": "Benchmarking Qwen3.8 27B quantizations: 4-bit holds up, 1-bit collapses"}]}, {"text": ", with ROCm crashes and regressions often pushing users to the Vulkan build.", "cites": [{"id": 788296288, "item_id": 49268518, "url": "https://news.ycombinator.com/item?id=49268518", "by": "imrehg", "date": "2026-08-11", "stance": "recommend", "quote": "I guess overall it's the worst runtime I've seen so far, except for all the other runtimes out there... I'm a fan", "excerpt": "llama.cpp works pretty well for me on the Framework 13 laptop, but the current era of \"move fast, break things, rarely fix\" (sorry, that's how it feels), bites here quite a bit. Two examples: - https://github.com/ggml-org/llama.cpp/pull/25863 Someone's few lines change broke the native (ROCm) support for the AMD GPU inside Framework (and other integrated systems), and any rollback or proper fix is pending for almost ", "context": "Framework 13 laptop user with AMD GPU", "use_case": "local LLM inference on Framework 13", "switched_from": "", "story": "llama.cpp"}, {"id": 793858096, "item_id": 49616131, "url": "https://news.ycombinator.com/item?id=49616131", "by": "civvv", "date": "2026-09-08", "stance": "recommend", "quote": "so I switched to the Windows Vulkan build of llama.cpp instead, which worked reliably.", "excerpt": "I got Qwen3.8-27B running on my RX 9070 XT by setting up AMD\u2019s ROCm 7.14/TheRock runtime for gfx1201 and adding the required HIP/LLVM paths to Windows. The official llama.cpp ROCm build detected the card correctly, but kept crashing with 0xC0000005 while loading the model, so I switched to the Windows Vulkan build of llama.cpp instead, which worked reliably. I\u2019m running Qwen3.8-27B-Unleashed UD-Q3_K_XL, which is a ~1", "context": "RX 9070 XT user on Windows", "use_case": "running Qwen locally on AMD GPU", "switched_from": "llama.cpp ROCm build", "story": "Benchmarking Qwen3.8 27B quantizations: 4-bit holds up, 1-bit collapses"}, {"id": 776719776, "item_id": 48544986, "url": "https://news.ycombinator.com/item?id=48544986", "by": "porkloin", "date": "2026-06-15", "stance": "recommend", "quote": "llama.cpp vulkan (better performance than ROCM, though I'm keeping an eye on it in the future)", "excerpt": "I have good results with this setup: Hardware: - GPU: AMD 7900xtx, 24gb vram - CPU: AMD 5950x, AM4 - RAM: 64gb DDR4 3600 Software: - OS: Bazzite (atomic fedora - this machine is running Steam \"big picture\" mode on my TV when not in use for LLM tasks) - Virtualization: Podman Quadlets, which allows me to run container images as managed systemd units - Network: tailscale - Inference: llama.cpp vulkan (better performanc", "context": "", "use_case": "local LLM inference", "switched_from": "", "story": "Ask HN: Has anyone replaced Claude/GPT with a local model for daily coding?"}]}], "praise": [{"label": "Ollama alternative", "summary": "Many switched from Ollama, citing better speed, features and openness.", "quotes": [{"id": 790471296, "item_id": 49404456, "url": "https://news.ycombinator.com/item?id=49404456", "by": "zepearl", "date": "2026-08-22", "stance": "recommend", "quote": "I now absolutely love all the knobs that \"llama.cpp\" has to tune your hardware setup & your workload", "excerpt": "I know nothing about \"vLLM\" -> anything better compared to \"llama.cpp\"? I started with \"Ollama\" (precompiled version) and it worked and was good enough to understand the very basics. Then I downloaded the sourcecode of \"llama.cpp\", compiled it with specific compilation options for my GPUs (CUDA/nVidia using proprietary module on Gentoo Linux) & CPU (AMD), and the same model ran twice as fast -> since then I stuck wit", "context": "Gentoo Linux user with NVIDIA GPUs and AMD CPU", "use_case": "running local LLMs", "switched_from": "Ollama", "story": "Why your local LLM feels dumber than it is"}, {"id": 791773008, "item_id": 49485813, "url": "https://news.ycombinator.com/item?id=49485813", "by": "wolvoleo", "date": "2026-08-29", "stance": "recommend", "quote": "I use Llama.cpp and OpenAI-style API of course", "excerpt": "I use Llama.cpp and OpenAI-style API of course (who doesn't, unless you use Ollama perhaps?) Ollama sucks, they dropped ROCm support for my card out of the blue, and the Vulkan replacement wasn't ready. The developer just shrugged and didn't care about the people affected so I moved to llama-server. Also, here's some other reasons to avoid ollama: https://sleepingrobots.com/dreams/stop-using-ollama/ And really, ollam", "context": "runs Linux server with multiple GPUs", "use_case": "running local LLMs", "switched_from": "Ollama", "story": "Run Qwen3.8 27B locally: real numbers from my Mac Studio"}, {"id": 783500929, "item_id": 48968808, "url": "https://news.ycombinator.com/item?id=48968808", "by": "ollamer", "date": "2026-07-19", "stance": "recommend", "quote": "On llama.cpp I can get models like Qwen 35BA3B running partially on gpu/cpu with 40t/s on a laptop thanks to --n-cpu-moe", "excerpt": "https://github.com/ollama/ollama/issues/11772 A year and still no implementation for such a basic need as offloading MoE layers onto the CPU selectively. On llama.cpp I can get models like Qwen 35BA3B running partially on gpu/cpu with 40t/s on a laptop thanks to --n-cpu-moe but on this VC funded joke it would be simply unusable. I can't quite understand how you make a wrapper so much worse than the code you're rippin", "context": "", "use_case": "partial CPU/GPU MoE inference", "switched_from": "", "story": "Ollama: All Aboard Open Models"}]}, {"label": "Performance tuning", "summary": "Lots of knobs and local builds let users get much more speed from their hardware.", "quotes": [{"id": 790471296, "item_id": 49404456, "url": "https://news.ycombinator.com/item?id=49404456", "by": "zepearl", "date": "2026-08-22", "stance": "recommend", "quote": "I now absolutely love all the knobs that \"llama.cpp\" has to tune your hardware setup & your workload", "excerpt": "I know nothing about \"vLLM\" -> anything better compared to \"llama.cpp\"? I started with \"Ollama\" (precompiled version) and it worked and was good enough to understand the very basics. Then I downloaded the sourcecode of \"llama.cpp\", compiled it with specific compilation options for my GPUs (CUDA/nVidia using proprietary module on Gentoo Linux) & CPU (AMD), and the same model ran twice as fast -> since then I stuck wit", "context": "Gentoo Linux user with NVIDIA GPUs and AMD CPU", "use_case": "running local LLMs", "switched_from": "Ollama", "story": "Why your local LLM feels dumber than it is"}, {"id": 776947536, "item_id": 48559221, "url": "https://news.ycombinator.com/item?id=48559221", "by": "spockz", "date": "2026-06-16", "stance": "recommend", "quote": "found a lot of performance improvement over ollama by compiling llama.cpp and running with \u2014no-mmap and \u2014perf", "excerpt": "For what it is worth, I\u2019m on a similar machine. (9070XT,5900X) and found a lot of performance improvement over ollama by compiling llama.cpp and running with \u2014no-mmap and \u2014perf. The context is still quite small though. With online models I use contexts of at least 200k which is useful for longer running/more complicated commands. Locally I haven\u2019t gone much further than 8k. That is sufficient for small changes on sma", "context": "has a 9070XT and 5900X", "use_case": "local LLM inference", "switched_from": "ollama", "story": "Running local models is good now"}, {"id": 777398656, "item_id": 48587416, "url": "https://news.ycombinator.com/item?id=48587416", "by": "alexellisuk", "date": "2026-06-18", "stance": "recommend", "quote": "but for how we use it llama.cpp is still better for us.", "excerpt": "We did run vLLM on the 3090s \u2014 measured ~3 tok/s slower on generation for our single-to-few-user pattern, plus less flexibility on quant and slower startup (actual minutes vs single digit seconds). We may do more with it again in the future - there isn't unlimited time for us to tinker, I'm sharing our journey (so far) and reasoning. It's the right call for concurrent batched serving (barrkel's point downthread is sp", "context": "small team running local Qwen on 3090s", "use_case": "local LLM serving", "switched_from": "", "story": "Local Qwen isn't a worse Opus, it's a different tool"}]}, {"label": "Rich features", "summary": "Commenters cite reasoning budgets, grammars, structured outputs, memory mapping and benchmarking tools.", "quotes": [{"id": 789210432, "item_id": 49325652, "url": "https://news.ycombinator.com/item?id=49325652", "by": "cyanydeez", "date": "2026-08-16", "stance": "recommend", "quote": "--thinking-budget and --thinking-message is all you need in llamacpp to keep it progressing.", "excerpt": "--thinking-budget and --thinking-message is all you need in llamacpp to keep it progressing. the message can be some combination of tool calling, summarizing, etc. It's overthinking often is a bunch of recursion, so simply stopping t and redirecting is all you need to do. If someones building a harness for llamacpp, you can set this per message, so it's possible to dynamically control it by watching for the expansion", "context": "", "use_case": "running local models", "switched_from": "", "story": "Qwen 3.8 27B is excellent, but it defaults to overthinking things"}, {"id": 777788912, "item_id": 48611807, "url": "https://news.ycombinator.com/item?id=48611807", "by": "anonym29", "date": "2026-06-20", "stance": "recommend", "quote": "llama.cpp supports structured outputs, as do some more consumer-oriented tools that wrap like LM Studio.", "excerpt": "In that case, you don't want to be using Claude Code, which is more of a consumer product; you instead want to control the inference stack yourself. What you are looking for is structured output (you give the inference engine a JSON schema you define that the response must conform to) + a JSON schema validator that parses the output, checks if the response is valid JSON. If it is, you're good to go, if not, run the i", "context": "", "use_case": "structured output inference", "switched_from": "", "story": "Ask HN: What technique do you use to make Claude Code deterministic?"}, {"id": 776131344, "item_id": 48508209, "url": "https://news.ycombinator.com/item?id=48508209", "by": "Aurornis", "date": "2026-06-12", "stance": "recommend", "quote": "llama.cpp includes a tool specifically for benchmarking that will sweep the arguments for you", "excerpt": "> The benchmark prompt was: > Write a compact Python function that parses a unified diff and returns the changed file paths. Then explain two edge cases. > Each benchmark generated about 128 tokens. Generating 128 tokens is probably not enough for good benchmark results. MTP speedup depends on how often the predicted tokens are accepted. In my experience, the very early output has a higher acceptance rate, so short t", "context": "", "use_case": "benchmarking local models", "switched_from": "", "story": "How to Setup a Local Coding Agent on macOS"}]}, {"label": "Server and web UI", "summary": "llama-server provides a built-in web UI, OpenAI-style API and multi-model routing.", "quotes": [{"id": 794057472, "item_id": 49628592, "url": "https://news.ycombinator.com/item?id=49628592", "by": "tarruda", "date": "2026-09-09", "stance": "recommend", "quote": "llama.cpp is simply better with its awesome built-in webui, router and server APIs", "excerpt": "I tried dwarfstar when llama.cpp DSV4 support was still very weak, and while it worked, I didn't see anything that would make me want to stick with it vs llama.cpp. llama.cpp is simply better with its awesome built-in webui, router and server APIs and certainly support much more models and quantizations than dwarfstar. Since then, I started maintaining my own vibe coded dsv4 branch with metal optimizations, so I actu", "context": "", "use_case": "running DeepSeek V4 locally", "switched_from": "dwarfstar", "story": "DeepSeek launching v4.1 flash cheaper and more capable than v4 pro"}, {"id": 800412368, "item_id": 50025773, "url": "https://news.ycombinator.com/item?id=50025773", "by": "genpfault", "date": "2026-10-09", "stance": "recommend", "quote": "llama-server's supported multiple models for a while now", "excerpt": "> https://github.com/darshi1337/apogee/blob/main/MODELS.md#local-llamacpp-models > Unlike Ollama, [llama.cpp] serves one model at a time: the GGUF you launched it with. FWIW: llama-server's supported multiple models for a while now: https://github.com/ggml-org/llama.cpp/tree/master/tools/server#using-multiple-models", "context": "", "use_case": "local LLM serving", "switched_from": "", "story": "Show HN: Apogee: Rebuilding Mozilla's Orbit, fully local and private"}, {"id": 782202704, "item_id": 48887669, "url": "https://news.ycombinator.com/item?id=48887669", "by": "MyMemoryfails", "date": "2026-07-12", "stance": "recommend", "quote": "Running Local LLM isn't really a lot effort anymore, since its one command away", "excerpt": "Running Local LLM isn't really a lot effort anymore, since its one command away to get the familar interface of chat AI. This setup requires 24GB ish VRAM and 32GB ram, so if you have capable gaming PC which can do AAA games, you can run it. - llama-server -hf ggml-org/gemma-4-26b-a4b-it-GGUF:Q4_K_M After that simply open browser and enter: http://localhost:8080 What this does: This will download Gemma4 AI with 26B p", "context": "", "use_case": "running local LLMs", "switched_from": "", "story": "Ask HN: How do you use LLMs for private discussions?"}]}, {"label": "Hardware support", "summary": "Works across Vulkan, ROCm, CUDA, SYCL and Apple hardware, including pooling VRAM across cards.", "quotes": [{"id": 798446130, "item_id": 49902883, "url": "https://news.ycombinator.com/item?id=49902883", "by": "latentsea", "date": "2026-09-30", "stance": "recommend", "quote": "Using llama.cpp Vulkan let's you pool VRAM across cards", "excerpt": "Qwen3.8-27B is a huge step up from Qwen3.6-27B. That release only happened relatively recently but that felt like the 'Opus 4.5' release turning point that SOTA models experienced back when that came out. It was the first time I felt like local models are actually good enough to use as daily drivers now. So, it was only after that point that I switched. Qwen3.8-Flash-Next is better still if you can run fast enough. I", "context": "", "use_case": "local inference across GPUs", "switched_from": "", "story": "GPT 6.1 Sol: Near-Astra intelligence for a fifth of the price"}, {"id": 788644961, "item_id": 49290310, "url": "https://news.ycombinator.com/item?id=49290310", "by": "SwellJoe", "date": "2026-08-13", "stance": "recommend", "quote": "they currently work great with llama.cpp and current ROCm 7.14", "excerpt": "\"used to\" in your sentence already answers your curiosity. ROCm runs most things just fine without any ceremony or difficulty. I have a couple of the same cards as covered in the article, bought before they got expensive (I'd recommend current Radeon AI Pro 9700 cards over the V620 now, though, as they have increased in price since I bought mine), and they're at the long end of the supported chart for ROCm...nearly E", "context": "", "use_case": "local inference", "switched_from": "", "story": "We Have AI at Home Chapter 1: A Box of Scraps"}, {"id": 782281553, "item_id": 48892597, "url": "https://news.ycombinator.com/item?id=48892597", "by": "kamranjon", "date": "2026-07-13", "stance": "recommend", "quote": "llama.cpp is pretty exciting right now with the SYCL improvements", "excerpt": "Some of the most exciting engineering work is happening in the DS4 repo - and I'm watching it almost like a sports game. When the DSpark paper came out[1] the next day we had folks attempting to implement, working together, validating their failures. Eventually their work being synthesized into a PR[1] that admits performance is not ideal. Something antirez alluded to in one of his videos (speculative decoding is a g", "context": "", "use_case": "local inference", "switched_from": "", "story": "Control the Ideas, Not the Code"}]}, {"label": "Active community", "summary": "Maintainers and contributors add new models and fixes quickly.", "quotes": [{"id": 793100720, "item_id": 49568795, "url": "https://news.ycombinator.com/item?id=49568795", "by": "kamranjon", "date": "2026-09-04", "stance": "recommend", "quote": "I use either llama.cpp itself or projects that depend on it all the time", "excerpt": "I follow llama.cpp pretty closely as I use either llama.cpp itself or projects that depend on it all the time, and one thing that I don't think gets talked about is the sheer scale of community involvement. It seems like a logistical nightmare, but somehow thousands of different contributors are opening hundreds of PR's every week and getting them merged in to support various hardware or implement a new pattern or al", "context": "", "use_case": "running local LLMs", "switched_from": "", "story": "Georgi Gerganov on llama.cpp/ggml future after Nvidia acquisition of HuggingFace"}, {"id": 788296000, "item_id": 49268500, "url": "https://news.ycombinator.com/item?id=49268500", "by": "karimf", "date": "2026-08-11", "stance": "recommend", "quote": "I highly recommend using llama.cpp for running AI model locally vs using other inference framework", "excerpt": "Not sure why it's on the front page now, but I highly recommend using llama.cpp for running AI model locally vs using other inference framework, unless you have a very specific requirement. ggerganov and the team have done a stellar job maintaining the quality while still being fast to implement new models/improvements.", "context": "", "use_case": "running AI models locally", "switched_from": "", "story": "llama.cpp"}, {"id": 793113472, "item_id": 49569592, "url": "https://news.ycombinator.com/item?id=49569592", "by": "springtimesun", "date": "2026-09-04", "stance": "recommend", "quote": "It does seem like llama.cpp is successfully riding that dragon right now though.", "excerpt": "I have been using a PR branch to run GLM 5.3 flash locally so I have been watching the dueling PRs develop to implement it and all the comments and reviews. You're right the scale of it is huge. The speed with which maintainers and community are responding is also super impressive. I know there's a lot of discussion about AI overwhelming OS maintainers and I can see that. It does seem like llama.cpp is successfully r", "context": "", "use_case": "running local models", "switched_from": "", "story": "Georgi Gerganov on llama.cpp/ggml future after Nvidia acquisition of HuggingFace"}]}], "caveats": [{"label": "GPU setup", "summary": "Getting the GPU used can take fiddling, and one user couldn't get it working on an Intel Arc card.", "quotes": [{"id": 776006098, "item_id": 48500381, "url": "https://news.ycombinator.com/item?id=48500381", "by": "dlcarrier", "date": "2026-06-11", "stance": "warn", "quote": "I can't get Llama.cpp to use the GPU", "excerpt": "I have an Intel A770, because at $200 USD it was the second cheapest way to get 16 of dedicated VRAM. (The cheapest is a $100 Radeon Instinct Mi25, but it's a bit old and slow.) I battled with Python packages a lot to get a1111 working, gave up when it broke on an update and switched to ComfyUI which has worked great. I can't get Llama.cpp to use the GPU, but I'm currently installing some of Intel's proprietary OpenV", "context": "", "use_case": "LLM on Intel GPU", "switched_from": "", "story": "Ask HN: What computer are you using for AI coding tools?"}, {"id": 784331137, "item_id": 49020696, "url": "https://news.ycombinator.com/item?id=49020696", "by": "gbuk2013", "date": "2026-07-23", "stance": "recommend", "quote": "there was some fiddling required to get it to use GPU (I use llama.cpp)", "excerpt": "I run Qwen3-Coder-Next-UD-Q4_K_XL and other than the initial wait to initialise context (which takes less than 2 minutes) subsequent prompts return in less than a minute, usually less than 30s. If your performance is significantly slower then you are probably doing it in CPU - there was some fiddling required to get it to use GPU (I use llama.cpp)", "context": "", "use_case": "local LLM inference", "switched_from": "", "story": "New Framework Desktop Option with AMD Ryzen AI Max+ Pro 495 and 192GB Memory"}]}, {"label": "ROCm breakage", "summary": "Users report ROCm crashes and regressions that remain unfixed, with Vulkan as the workaround.", "quotes": [{"id": 788296288, "item_id": 49268518, "url": "https://news.ycombinator.com/item?id=49268518", "by": "imrehg", "date": "2026-08-11", "stance": "recommend", "quote": "I guess overall it's the worst runtime I've seen so far, except for all the other runtimes out there... I'm a fan", "excerpt": "llama.cpp works pretty well for me on the Framework 13 laptop, but the current era of \"move fast, break things, rarely fix\" (sorry, that's how it feels), bites here quite a bit. Two examples: - https://github.com/ggml-org/llama.cpp/pull/25863 Someone's few lines change broke the native (ROCm) support for the AMD GPU inside Framework (and other integrated systems), and any rollback or proper fix is pending for almost ", "context": "Framework 13 laptop user with AMD GPU", "use_case": "local LLM inference on Framework 13", "switched_from": "", "story": "llama.cpp"}, {"id": 793858096, "item_id": 49616131, "url": "https://news.ycombinator.com/item?id=49616131", "by": "civvv", "date": "2026-09-08", "stance": "recommend", "quote": "so I switched to the Windows Vulkan build of llama.cpp instead, which worked reliably.", "excerpt": "I got Qwen3.8-27B running on my RX 9070 XT by setting up AMD\u2019s ROCm 7.14/TheRock runtime for gfx1201 and adding the required HIP/LLVM paths to Windows. The official llama.cpp ROCm build detected the card correctly, but kept crashing with 0xC0000005 while loading the model, so I switched to the Windows Vulkan build of llama.cpp instead, which worked reliably. I\u2019m running Qwen3.8-27B-Unleashed UD-Q3_K_XL, which is a ~1", "context": "RX 9070 XT user on Windows", "use_case": "running Qwen locally on AMD GPU", "switched_from": "llama.cpp ROCm build", "story": "Benchmarking Qwen3.8 27B quantizations: 4-bit holds up, 1-bit collapses"}]}, {"label": "Fast-moving releases", "summary": "Users rebuild often to keep up, and changes sometimes break things.", "quotes": [{"id": 784233680, "item_id": 49014605, "url": "https://news.ycombinator.com/item?id=49014605", "by": "hedgehog", "date": "2026-07-22", "stance": "recommend", "quote": "I recommend building recent llama.cpp from source, I've been updating about once a week", "excerpt": "Maybe enabling MTP causes some weights to be displaced to host memory? MTP itself doesn't do anything during prefill so that should be exactly unchanged, decode will vary depending on settings but with 2-4 proposals depending on workload I've never seen an overall slowdown. edit: I recommend building recent llama.cpp from source, I've been updating about once a week, as there has been a fair amount of work related to", "context": "", "use_case": "local LLM inference", "switched_from": "", "story": "Nvidia DGX Spark as a daily driver"}, {"id": 788296288, "item_id": 49268518, "url": "https://news.ycombinator.com/item?id=49268518", "by": "imrehg", "date": "2026-08-11", "stance": "recommend", "quote": "I guess overall it's the worst runtime I've seen so far, except for all the other runtimes out there... I'm a fan", "excerpt": "llama.cpp works pretty well for me on the Framework 13 laptop, but the current era of \"move fast, break things, rarely fix\" (sorry, that's how it feels), bites here quite a bit. Two examples: - https://github.com/ggml-org/llama.cpp/pull/25863 Someone's few lines change broke the native (ROCm) support for the AMD GPU inside Framework (and other integrated systems), and any rollback or proper fix is pending for almost ", "context": "Framework 13 laptop user with AMD GPU", "use_case": "local LLM inference on Framework 13", "switched_from": "", "story": "llama.cpp"}]}, {"label": "Harder setup", "summary": "Harder to set up than simpler wrappers, though more performant.", "quotes": [{"id": 793011681, "item_id": 49563230, "url": "https://news.ycombinator.com/item?id=49563230", "by": "KronisLV", "date": "2026-09-04", "stance": "recommend", "quote": "there's llama.cpp and vLLM, each harder to setup but opens the door for squeezing more performance out of your hardware", "excerpt": "> Download ollama and give it a try Weren't people pissed off at them for a variety of reasons? Not giving attribution etc. In case anyone wants other good alternatives, there's llama.cpp and vLLM, each harder to setup but opens the door for squeezing more performance out of your hardware. I've also heard okay things about LM Studio and I think Unsloth had their own thing as well: https://unsloth.ai/docs/new/studio >", "context": "", "use_case": "running local LLMs", "switched_from": "", "story": "Which tools do Claude, Codex and Cursor choose? We measured 17k runs to find out"}]}], "who": [{"by": "testerius", "context": "hobbyist with old GPU", "url": "https://news.ycombinator.com/item?id=50129389", "date": "2026-10-17"}, {"by": "lxe", "context": "runs a local inference box", "url": "https://news.ycombinator.com/item?id=49912394", "date": "2026-09-30"}, {"by": "LeBit", "context": "Mac Mini with 48G", "url": "https://news.ycombinator.com/item?id=49836240", "date": "2026-09-24"}, {"by": "civvv", "context": "RX 9070 XT user on Windows", "url": "https://news.ycombinator.com/item?id=49616131", "date": "2026-09-08"}, {"by": "jwr", "context": "64GB MacBook Pro M4 Max user", "url": "https://news.ycombinator.com/item?id=49533837", "date": "2026-09-02"}, {"by": "wolvoleo", "context": "runs Linux server with multiple GPUs", "url": "https://news.ycombinator.com/item?id=49485813", "date": "2026-08-29"}, {"by": "cyanydeez", "context": "homelab user", "url": "https://news.ycombinator.com/item?id=49449055", "date": "2026-08-26"}, {"by": "SwellJoe", "context": "runs local LLMs on AMD Strix Halo", "url": "https://news.ycombinator.com/item?id=49437348", "date": "2026-08-25"}, {"by": "zepearl", "context": "Gentoo Linux user with NVIDIA GPUs and AMD CPU", "url": "https://news.ycombinator.com/item?id=49404456", "date": "2026-08-22"}, {"by": "mirekrusin", "context": "dual 4090 setup", "url": "https://news.ycombinator.com/item?id=49339498", "date": "2026-08-17"}, {"by": "sgt", "context": "RTX 5090 owner", "url": "https://news.ycombinator.com/item?id=49310844", "date": "2026-08-15"}, {"by": "imrehg", "context": "Framework 13 laptop user with AMD GPU", "url": "https://news.ycombinator.com/item?id=49268518", "date": "2026-08-11"}], "tags": ["for local LLM inference", "for AMD GPU users", "for Mac users", "for local coding agents", "for home servers"], "use_cases": ["running local models", "local llm inference", "local inference", "running local llms", "serving local models", "local model inference"], "compared": [{"slug": "ollama", "name": "Ollama", "co": 31, "switched_from": 14, "switched_to": 0}, {"slug": "qwen-3-6", "name": "Qwen 3.6", "co": 35, "switched_from": 0, "switched_to": 0}, {"slug": "vllm", "name": "vLLM", "co": 17, "switched_from": 2, "switched_to": 1}, {"slug": "opencode", "name": "OpenCode", "co": 21, "switched_from": 0, "switched_to": 0}, {"slug": "lm-studio", "name": "LM Studio", "co": 17, "switched_from": 1, "switched_to": 1}, {"slug": "qwen-3-5", "name": "Qwen 3.5", "co": 15, "switched_from": 0, "switched_to": 0}, {"slug": "qwen-3-8", "name": "Qwen 3.8", "co": 15, "switched_from": 0, "switched_to": 0}, {"slug": "pi", "name": "Pi", "co": 15, "switched_from": 0, "switched_to": 0}], "makers": [{"id": 799307200, "item_id": 49956700, "url": "https://news.ycombinator.com/item?id=49956700", "by": "amankarki", "date": "2026-10-04", "stance": "neutral", "quote": "2 CUDA PRs merged into llama.cpp", "excerpt": "Location: India (IST, GMT+5:30) Remote: Yes Willing to relocate: No Technologies: C++20, CUDA, CUTLASS, Triton, Python, PyTorch, vLLM, llama.cpp/ggml, FP8/INT8 quantization, Linux R\u00e9sum\u00e9/CV: https://aman-portfolio-rho-six.vercel.app/resume.pdf Email: itsamankarki@gmail.com I write C++ and CUDA for LLM inference. 2 CUDA PRs merged into llama.cpp; now working on block-FP8 GEMM support for Ada GPUs (L4/L40S/4090) in vLL", "context": "C++/CUDA engineer for LLM inference", "use_case": "LLM inference contributions", "switched_from": "", "story": "Ask HN: Who wants to be hired? (October 2026)"}, {"id": 799056416, "item_id": 49941026, "url": "https://news.ycombinator.com/item?id=49941026", "by": "xlayn", "date": "2026-10-03", "stance": "recommend", "quote": "I keep this branch of llama.cpp that includes that same functionality", "excerpt": "In case you like the store kv to disk so you can resume I keep this branch of llama.cpp that includes that same functionality https://github.com/alainnothere/llama.cpp/commits/disk-cache-eviction And you know it's load bearing each of the load baerings parts that bear some load and load a bear... you fight a bear because it took a load... or something like that...", "context": "", "use_case": "KV cache to disk", "switched_from": "", "story": "From the creator of Redis; run LLM locally with ds4"}, {"id": 789688177, "item_id": 49355511, "url": "https://news.ycombinator.com/item?id=49355511", "by": "xlayn", "date": "2026-08-18", "stance": "neutral", "quote": "you have to use my version of llama.cpp in the link to try that", "excerpt": "Because it's so nice to see page after page after page of... but what if... let's consider... If you use a gguf, you can extract the template from the Qwen3.827B model using the following script https://github.com/alainnothere/llama.cpp/blob/disk-cache-eviction/scripts/gguf-get-chat-template.py, use it like ggufGetTemplate.py MODEL.gguf [OUT.jinja] modify the template to your liking, and then use https://github.com/a", "context": "", "use_case": "running local models", "switched_from": "", "story": "Qwen3.8-27B make medium the default effort level instead of xhigh"}, {"id": 789434817, "item_id": 49339676, "url": "https://news.ycombinator.com/item?id=49339676", "by": "xlayn", "date": "2026-08-17", "stance": "recommend", "quote": "It's been tested 99% of the time against a local llama.cpp", "excerpt": "I've been working on this clone of mistral vibe, the idea is to have a cli that you can trust is not calling back home for any reason, I have done as much work as possible to remove any trace of phone back anywhere, hence the pri prefix to the name privibe. It's been tested 99% of the time against a local llama.cpp, I have also used it against open router with no issues. Then on top of it, I want to be something priv", "context": "", "use_case": "local model serving", "switched_from": "", "story": "Privibe \u2013 LLM CLI Local first+privacy focus+llama.cpp cache branch and Qwen3.x"}, {"id": 789380432, "item_id": 49336277, "url": "https://news.ycombinator.com/item?id=49336277", "by": "ggerganov", "date": "2026-08-17", "stance": "neutral", "quote": "We are preparing official semantic versioning of llama.cpp", "excerpt": "Didn't expect this to pop up here - please ignore for now. We are preparing official semantic versioning of llama.cpp and it's almost ready, but not quite. More info about the versioning process is here: https://github.com/ggml-org/ggml/discussions/1579", "context": "", "use_case": "", "switched_from": "", "story": "Llama.cpp v0.1.0"}, {"id": 789240864, "item_id": 49327554, "url": "https://news.ycombinator.com/item?id=49327554", "by": "hellajack3d", "date": "2026-08-17", "stance": "neutral", "quote": "I forked llama.cpp and added some crude mechanism to keep exactly this behavior under control", "excerpt": "I forked llama.cpp and added some crude mechanism to keep exactly this behavior under control - essentially guiding the reasoning process by injecting text strategically at specific thresholds. This was mainly put together to rein in Qwen3.6-27B, but I'd imagine 3.8 would react similarly. Fork can be found here - https://github.com/laurencehardman/llama-mindcontrol/tree/master Of course hacks like this are not perfec", "context": "", "use_case": "controlling reasoning of local models", "switched_from": "", "story": "Qwen 3.8 27B is excellent, but it defaults to overthinking things"}], "threads": [{"url": "https://news.ycombinator.com/item?id=48542100", "title": "Ask HN: Has anyone replaced Claude/GPT with a local model for daily coding?", "mentions": 16}, {"url": "https://news.ycombinator.com/item?id=49267928", "title": "llama.cpp", "mentions": 9}, {"url": "https://news.ycombinator.com/item?id=48555993", "title": "Running local models is good now", "mentions": 9}, {"url": "https://news.ycombinator.com/item?id=49299605", "title": "Qwen 3.8 27B is out: open weights, best local dense model yet", "mentions": 5}, {"url": "https://news.ycombinator.com/item?id=49324985", "title": "Qwen 3.8 27B is excellent, but it defaults to overthinking things", "mentions": 4}, {"url": "https://news.ycombinator.com/item?id=45771870", "title": "Ask HN: Who uses open LLMs and coding assistants locally? Share setup and laptop", "mentions": 3}], "sentiment_by_month": {"2025-11": {"recommend": 2}, "2025-12": {"recommend": 2}, "2026-01": {"neutral": 2}, "2026-02": {"recommend": 2}, "2026-03": {"neutral": 3, "recommend": 1}, "2026-04": {"recommend": 4, "warn": 1, "neutral": 6}, "2026-05": {"recommend": 4, "neutral": 2, "warn": 2}, "2026-06": {"warn": 2, "recommend": 16, "neutral": 20, "compare": 1}, "2026-07": {"neutral": 7, "recommend": 7}, "2026-08": {"recommend": 17, "compare": 3, "neutral": 14}, "2026-09": {"recommend": 11, "neutral": 10, "compare": 2}, "2026-10": {"recommend": 2, "neutral": 5}}, "quotes": [{"id": 802668672, "item_id": 50166792, "url": "https://news.ycombinator.com/item?id=50166792", "by": "mceleri", "date": "2026-10-20", "stance": "neutral", "quote": "32 parallel sequences for your llamacpp (or similar)", "excerpt": "I love local AI, but it just can't compete with frontier models (unless your \"local\" is something similar to a home datacenter). \"Let's assume the bulk of your one-hour session was with an average of 32 background agents doing different tasks, at an average of 20k tok/s input\" Then you do need 32 parallel sequences for your llamacpp (or similar), 20k context (which can be a bit small for a coding agent) * 32 parallel", "context": "", "use_case": "local LLM inference", "switched_from": "", "story": "Agent Swarms Are Great for Local AI"}, {"id": 802504224, "item_id": 50156514, "url": "https://news.ycombinator.com/item?id=50156514", "by": "SwellJoe", "date": "2026-10-19", "stance": "neutral", "quote": "many Qwen-derived models run fine without any changes to llama.cpp", "excerpt": "Yes, that's apparent now. But, many Qwen-derived models run fine without any changes to llama.cpp, and there are gguf quantizations on HuggingFace, so I thought I'd give it a try.", "context": "", "use_case": "running local models", "switched_from": "", "story": "Hybrid Reasoning Agents: LLM agents that think in latent space, act through text"}, {"id": 802070224, "item_id": 50129389, "url": "https://news.ycombinator.com/item?id=50129389", "by": "testerius", "date": "2026-10-17", "stance": "neutral", "quote": "I tried llama.cpp with offloading \"all what could be\" to my GPU", "excerpt": "It sounds great and I hope local LLM/AI will be more powerful and available even on old hardware. I tried llama.cpp with offloading \"all what could be\" to my GPU (GTX 750 Ti 2 GB). I could run local Qwen2.5-Coder quite fast (I do not remember but something like 25-30 t/s). However this model generates bad code (I tried Python and JavaScript). So, for my setup it is better to chat with free ChatGPT and do copying-past", "context": "hobbyist with old GPU", "use_case": "local LLM inference", "switched_from": "", "story": "Show HN: Llmspeed \u2013 feel how fast a local LLM runs before you buy the hardware"}, {"id": 801161123, "item_id": 50072570, "url": "https://news.ycombinator.com/item?id=50072570", "by": "testerius", "date": "2026-10-13", "stance": "neutral", "quote": "it works fine with offloading to GPU and using llama.cpp", "excerpt": "I use Claude Opus 5.5 and Codex Sol 6.1 in my job and I love it. I wish be able to run local models which have power at least current Opus 5.5 on medium hardware which does not cost millions $. This is my dream, running LLMs and AI agents to just create things for myself. On my really old hardware I can only run simple and small 1.5B model (Qwen2.5-Coder-1.5B) and well... Ihmo it works fine with offloading to GPU and", "context": "runs on old hardware", "use_case": "running local models", "switched_from": "", "story": "Intel D270 GPU AI Accelerator\u2013Up to 480 GB LPDDR5X"}, {"id": 800412368, "item_id": 50025773, "url": "https://news.ycombinator.com/item?id=50025773", "by": "genpfault", "date": "2026-10-09", "stance": "recommend", "quote": "llama-server's supported multiple models for a while now", "excerpt": "> https://github.com/darshi1337/apogee/blob/main/MODELS.md#local-llamacpp-models > Unlike Ollama, [llama.cpp] serves one model at a time: the GGUF you launched it with. FWIW: llama-server's supported multiple models for a while now: https://github.com/ggml-org/llama.cpp/tree/master/tools/server#using-multiple-models", "context": "", "use_case": "local LLM serving", "switched_from": "", "story": "Show HN: Apogee: Rebuilding Mozilla's Orbit, fully local and private"}, {"id": 800375877, "item_id": 50023492, "url": "https://news.ycombinator.com/item?id=50023492", "by": "deniska", "date": "2026-10-09", "stance": "neutral", "quote": "llama.cpp serving gemma4-26b-a4b for image processing of things too sensitive to let openai and other similar companies to see it", "excerpt": "It's not exactly a cheap VPS, it's a server with 376 gb of ram my dad bought on ebay, put on co-location, and I help to administer it, and can create some VMs on it. But some of the things I do on these VMs are in principle similar to what I would do a $5 vps: - irssi running in screen (irc client, yes, occasionally I still talk to people on there) - my very small imgur-inspired image hosting (https://github.com/deni", "context": "", "use_case": "local image processing", "switched_from": "", "story": "Ask HN: What do you run on a $5 VPS that's worth keeping online 24/7?"}, {"id": 798924707, "item_id": 49932794, "url": "https://news.ycombinator.com/item?id=49932794", "by": "nfriedly", "date": "2026-10-02", "stance": "neutral", "quote": "these days I've figure out llama.cpp and proper context sizes", "excerpt": "I really like Pi, especially for when you're running a local model and your context is particularly limited. But even when running a big model, I think building up your environment to be exactly what you want can be really powerful. When I was first getting started and didn't realize that ollama defaults to incredibly tiny amounts of context, Pi was the only harness that actually worked because the rest of them used ", "context": "", "use_case": "running local models", "switched_from": "Ollama", "story": "Pi 1.0"}, {"id": 798837844, "item_id": 49927365, "url": "https://news.ycombinator.com/item?id=49927365", "by": "razster", "date": "2026-10-01", "stance": "recommend", "quote": "one tab is GitBash running llama.ccp and the other is normal terminal running pi. Winning combo for me.", "excerpt": "I use my Pi to organize my graphic design assets as well as update documentation for client projects. It also is my daily driver for a story I'm working on and thanks to a ton of great skills created it has become more powerful. Local model is Qwen3.6 35B-A3B and Qwen3.8 27B UD. I love the compressing function, keeps on pushing. I've tried Oh-my-pi but it's too heavy for me and eats up 6% of my context on start. Also", "context": "", "use_case": "running local models", "switched_from": "", "story": "Pi 1.0"}, {"id": 798599233, "item_id": 49912452, "url": "https://news.ycombinator.com/item?id=49912452", "by": "anerli", "date": "2026-09-30", "stance": "compare", "quote": "Since it seems llama.cpp did not optimize that path, we used 16-bit KV instead.", "excerpt": "The benchmark we cited here is a simple prose-repetition task. We put the content of Moby Dick up to 64k context in the request, and then ask it to repeat the last section. For llama.cpp, we try to make the comparison as fair as possible by using similar settings. No speculative decoding, default prefill batch sizes, flash attention on. We tried also quantizing the KV cache to 8-bit keys and 4-bit values like we do i", "context": "", "use_case": "benchmark baseline", "switched_from": "", "story": "Launch HN: Magnitude (YC S25) \u2013 Self-optimizing inference engine for agents"}, {"id": 798598305, "item_id": 49912394, "url": "https://news.ycombinator.com/item?id=49912394", "by": "lxe", "date": "2026-09-30", "stance": "neutral", "quote": "Then it rebuilds latest llama.cpp, grabs the PRs it finds relevant to test against", "excerpt": "On my local inference box I have a perpetual codex thread open in my llama.cpp checkout that I periodically ask to take a look at currently pending llama.cpp PRs, do some research on latest MTP, Dflash and other prediction or attention optimizations, do research on the latest model quants and finetunes, take a look at localLlama Reddit threads and just do essentially a sweep of the frontier. Then it rebuilds latest l", "context": "runs a local inference box", "use_case": "local inference", "switched_from": "", "story": "Launch HN: Magnitude (YC S25) \u2013 Self-optimizing inference engine for agents"}, {"id": 798446130, "item_id": 49902883, "url": "https://news.ycombinator.com/item?id=49902883", "by": "latentsea", "date": "2026-09-30", "stance": "recommend", "quote": "Using llama.cpp Vulkan let's you pool VRAM across cards", "excerpt": "Qwen3.8-27B is a huge step up from Qwen3.6-27B. That release only happened relatively recently but that felt like the 'Opus 4.5' release turning point that SOTA models experienced back when that came out. It was the first time I felt like local models are actually good enough to use as daily drivers now. So, it was only after that point that I switched. Qwen3.8-Flash-Next is better still if you can run fast enough. I", "context": "", "use_case": "local inference across GPUs", "switched_from": "", "story": "GPT 6.1 Sol: Near-Astra intelligence for a fifth of the price"}, {"id": 798016000, "item_id": 49876000, "url": "https://news.ycombinator.com/item?id=49876000", "by": "busfahrer", "date": "2026-09-28", "stance": "recommend", "quote": "I use this solution for your exact use case", "excerpt": "I use this solution for your exact use case: I have a single command that fires up llama.cpp on cpu only using gemma4 e2b, answers a single question from the command line and exits. This takes about 3 seconds to load from an SSD, and is smart enough to solve exactly these \"remind me of the syntax\" scenarios if you dont wanna switch to a browser.", "context": "", "use_case": "CPU-only command syntax lookup", "switched_from": "", "story": "Ember-1"}, {"id": 797970624, "item_id": 49873164, "url": "https://news.ycombinator.com/item?id=49873164", "by": "jadidbourbaki", "date": "2026-09-28", "stance": "neutral", "quote": "Lemire absolutely crushed it! I benchmarked his change today.", "excerpt": "As an update to this, Lemire absolutely crushed it! I benchmarked his change today. Here is my PR comment: https://github.com/jadidbourbaki/llama.cpp/pull/12#issuecomment-5856228676", "context": "", "use_case": "benchmarking prompt lookup drafting", "switched_from": "", "story": "Faster prompt lookup drafting in llama.cpp"}, {"id": 797546881, "item_id": 49846680, "url": "https://news.ycombinator.com/item?id=49846680", "by": "overtone1000", "date": "2026-09-25", "stance": "neutral", "quote": "That's what I've done when playing around with llama.", "excerpt": "When I just want to make something work and there isn't a nix package that can instantly give me what I want with `nix-shell -p $PACKAGE` then I use containers. That's what I've done when playing around with llama.", "context": "", "use_case": "running local LLMs", "switched_from": "", "story": "Dutch governments builds alternative for Microsoft based on NixOS"}, {"id": 797379843, "item_id": 49836240, "url": "https://news.ycombinator.com/item?id=49836240", "by": "LeBit", "date": "2026-09-24", "stance": "recommend", "quote": "llama.cpp + llama-swap", "excerpt": "I use the following models: - Qwen3.5 9B - Qwen3.6 35B A3B - Qwen3.8 27B - Gemma 4 26B A4B - Gemma 31B - Muse Glimmer 30B I have 48G. The MacMini is used solely for inference. llama.cpp + llama-swap. People are saying local models are crap and serves no purposes. I use them to help me spell check, write emails, write text messages, write JIRA tickets, write PR comments, etc. I also use them in coding agents to comple", "context": "Mac Mini with 48G", "use_case": "local model inference", "switched_from": "", "story": "Best LLM for every budget, updated daily"}, {"id": 797135248, "item_id": 49820953, "url": "https://news.ycombinator.com/item?id=49820953", "by": "grigio", "date": "2026-09-23", "stance": "neutral", "quote": "llama.cpp has a web ui nowdays", "excerpt": "llama.cpp has a web ui nowdays, but with an harness is easier to interact with existing cli tools", "context": "", "use_case": "running local LLMs", "switched_from": "", "story": "Ask HN: Which harness do you use with Qwen3.8 27B? And why?"}, {"id": 796459264, "item_id": 49778704, "url": "https://news.ycombinator.com/item?id=49778704", "by": "nperez", "date": "2026-09-20", "stance": "recommend", "quote": "I use a custom fork of llama.cpp that has an abliteration feature", "excerpt": "Yeah I use a custom fork of llama.cpp that has an abliteration feature that basically does this. It's sloppily vibe coded and I don't have time to coordinate on a way to do this cleanly upstream, but it's absolutely possible and saves a lot of time and bandwidth from being wasted", "context": "", "use_case": "local LLM inference with abliteration", "switched_from": "", "story": "Pirate Face Rescues LLM Models from Deletion"}, {"id": 795958241, "item_id": 49747390, "url": "https://news.ycombinator.com/item?id=49747390", "by": "simonw", "date": "2026-09-17", "stance": "neutral", "quote": "you need Prism's llama.cpp fork to get them to work", "excerpt": "If you want to try out out the GGUFs from https://huggingface.co/prism-ml/Ternary-Bonsai-2-27B-gguf#these-files-need-our-llamacpp-build be aware that you need Prism's llama.cpp fork to get them to work, from https://github.com/PrismML-Eng/llama.cpp/releases/tag/prism-b10685-7dffb15 This should work: cd /tmp # Get the Prism macOS runtime curl -fL https://github.com/PrismML-Eng/llama.cpp/releases/download/prism-b10685-", "context": "", "use_case": "running GGUF models", "switched_from": "", "story": "Bonsai 2 27B: Near-Lossless Compression in a 9x Smaller Footprint"}, {"id": 795259281, "item_id": 49703705, "url": "https://news.ycombinator.com/item?id=49703705", "by": "sbankowi", "date": "2026-09-14", "stance": "compare", "quote": "llama.cpp already makes recommendations on what models to run.", "excerpt": "I am looked at the website, but I am not able to understand what problem you are trying to solve. Or why would I use you? (How are you differentiating yourself?) I see you are reliant on ollama and llama.cpp (for example), so why should I not just use those directly if I am focused on local first models. (llama.cpp already makes recommendations on what models to run.) If the argument is then that you are more of an a", "context": "", "use_case": "running local models", "switched_from": "", "story": "Show HN: I built Otis, a minimal AI agent that runs local models out of the box"}, {"id": 795094193, "item_id": 49693387, "url": "https://news.ycombinator.com/item?id=49693387", "by": "thread123", "date": "2026-09-14", "stance": "neutral", "quote": "later grew support for continuing conversations through llama.cpp or OpenRouter", "excerpt": "https://github.com/ChrystianSchutz/ThreadShelf I had a lot of chats in OpenRouter, google AI Studio and lm studio and no good way to view/search/combine them together. ThreadShelf started as a local archive with semantic search/MCP, and later grew support for continuing conversations through llama.cpp or OpenRouter. It is my first time creating oss so please be gentle.", "context": "", "use_case": "continuing conversations", "switched_from": "", "story": "Ask HN: What are you working on? (September 2026)"}, {"id": 794441872, "item_id": 49652617, "url": "https://news.ycombinator.com/item?id=49652617", "by": "cube00", "date": "2026-09-11", "stance": "recommend", "quote": "More ethical alternatives have caught up including llama.cpp", "excerpt": ">Why I use Ollama. To be frank it's just easy. More ethical alternatives have caught up including llama.cpp https://sleepingrobots.com/dreams/stop-using-ollama/#what-to-use-instead", "context": "", "use_case": "local LLM inference", "switched_from": "", "story": "Setting up OpenCode with Ollama and sbx on Mac"}, {"id": 794202773, "item_id": 49637673, "url": "https://news.ycombinator.com/item?id=49637673", "by": "softwarewright", "date": "2026-09-10", "stance": "neutral", "quote": "served by llama.cpp, Freetoken, or vLLM", "excerpt": "Claude Code, Codex, opencode (Z.ai GLM-5.x), sometimes Gemini -- most of my work is at Mac iTerm2 CLI or Arch Linux terminal. I sometimes use emacs as the coding agent, but rarely JetBrains IDE or vscode (more for reading/debugging code). I supplement this with my own vibe-coded tools that help agents plan, perform sagas/steps, stay on track, tools that check the code produced, tools that check the produced documenta", "context": "", "use_case": "serving local LLMs", "switched_from": "", "story": "Ask HN: What's your AI coding set up?"}, {"id": 794129633, "item_id": 49633102, "url": "https://news.ycombinator.com/item?id=49633102", "by": "spijdar", "date": "2026-09-09", "stance": "neutral", "quote": "done through llama.cpp's web UI", "excerpt": "This is Unsloth's UD-Q4_K_S quantization (edit -- on llama.cpp, via the Vulkan backend, on an RX 7900 XT, with Unsloth's recommended sampler config), for \"as replicable as LLMs can be\" disclosure, done through llama.cpp's web UI. In another thread, via Open WebUI (both with no custom prompt, so whatever the \"default\" system prompt is), a reasoning block opens as such: We need answer user's request: write complete sho", "context": "", "use_case": "running local models", "switched_from": "", "story": "Qwen 3.8 follows GPT-5.5 Pro reasoning prefills"}, {"id": 794057472, "item_id": 49628592, "url": "https://news.ycombinator.com/item?id=49628592", "by": "tarruda", "date": "2026-09-09", "stance": "recommend", "quote": "llama.cpp is simply better with its awesome built-in webui, router and server APIs", "excerpt": "I tried dwarfstar when llama.cpp DSV4 support was still very weak, and while it worked, I didn't see anything that would make me want to stick with it vs llama.cpp. llama.cpp is simply better with its awesome built-in webui, router and server APIs and certainly support much more models and quantizations than dwarfstar. Since then, I started maintaining my own vibe coded dsv4 branch with metal optimizations, so I actu", "context": "", "use_case": "running DeepSeek V4 locally", "switched_from": "dwarfstar", "story": "DeepSeek launching v4.1 flash cheaper and more capable than v4 pro"}, {"id": 794034641, "item_id": 49627165, "url": "https://news.ycombinator.com/item?id=49627165", "by": "tarruda", "date": "2026-09-09", "stance": "neutral", "quote": "llama.cpp master is still somewhat bad in Qwen 3.8 next performance", "excerpt": "Don't use my Vision-Exp GGUF though. As I said I built those GGUFs before llama.cpp supported, and they can't be loaded on current master (require my own branch). I already have new GGUFs but haven't uploaded yet. If you want Vision-Exp, maybe use bartowski or unsloth's GGUFs. Side note: As an alternative to deepseek v4, you might want to give it a shot at qwen 3.8 flash next. I have IQ4_NL GGUFs that can be loaded f", "context": "", "use_case": "local inference", "switched_from": "", "story": "DeepSeek launching v4.1 flash cheaper and more capable than v4 pro"}, {"id": 793858096, "item_id": 49616131, "url": "https://news.ycombinator.com/item?id=49616131", "by": "civvv", "date": "2026-09-08", "stance": "recommend", "quote": "so I switched to the Windows Vulkan build of llama.cpp instead, which worked reliably.", "excerpt": "I got Qwen3.8-27B running on my RX 9070 XT by setting up AMD\u2019s ROCm 7.14/TheRock runtime for gfx1201 and adding the required HIP/LLVM paths to Windows. The official llama.cpp ROCm build detected the card correctly, but kept crashing with 0xC0000005 while loading the model, so I switched to the Windows Vulkan build of llama.cpp instead, which worked reliably. I\u2019m running Qwen3.8-27B-Unleashed UD-Q3_K_XL, which is a ~1", "context": "RX 9070 XT user on Windows", "use_case": "running Qwen locally on AMD GPU", "switched_from": "llama.cpp ROCm build", "story": "Benchmarking Qwen3.8 27B quantizations: 4-bit holds up, 1-bit collapses"}, {"id": 793276513, "item_id": 49579782, "url": "https://news.ycombinator.com/item?id=49579782", "by": "Vibecoder_", "date": "2026-09-05", "stance": "neutral", "quote": "I keep ending up with the same problem when running llama.cpp", "excerpt": "I keep ending up with the same problem when running llama.cpp: multiple models, multiple instances, different GPU allocations, ports, flags, and a growing pile of scripts to keep it all manageable. LlamaRack is my attempt at making that less annoying. It manages llama.cpp instances, models, GPU assignment and runtime settings, with an OpenAI-compatible API on top. It works well enough on my own setup now, but I\u2019d rea", "context": "", "use_case": "running local models", "switched_from": "", "story": "Anyone using llama.cpp willing to test LlamaRack?"}, {"id": 793113472, "item_id": 49569592, "url": "https://news.ycombinator.com/item?id=49569592", "by": "springtimesun", "date": "2026-09-04", "stance": "recommend", "quote": "It does seem like llama.cpp is successfully riding that dragon right now though.", "excerpt": "I have been using a PR branch to run GLM 5.3 flash locally so I have been watching the dueling PRs develop to implement it and all the comments and reviews. You're right the scale of it is huge. The speed with which maintainers and community are responding is also super impressive. I know there's a lot of discussion about AI overwhelming OS maintainers and I can see that. It does seem like llama.cpp is successfully r", "context": "", "use_case": "running local models", "switched_from": "", "story": "Georgi Gerganov on llama.cpp/ggml future after Nvidia acquisition of HuggingFace"}, {"id": 793100720, "item_id": 49568795, "url": "https://news.ycombinator.com/item?id=49568795", "by": "kamranjon", "date": "2026-09-04", "stance": "recommend", "quote": "I use either llama.cpp itself or projects that depend on it all the time", "excerpt": "I follow llama.cpp pretty closely as I use either llama.cpp itself or projects that depend on it all the time, and one thing that I don't think gets talked about is the sheer scale of community involvement. It seems like a logistical nightmare, but somehow thousands of different contributors are opening hundreds of PR's every week and getting them merged in to support various hardware or implement a new pattern or al", "context": "", "use_case": "running local LLMs", "switched_from": "", "story": "Georgi Gerganov on llama.cpp/ggml future after Nvidia acquisition of HuggingFace"}, {"id": 793020256, "item_id": 49563766, "url": "https://news.ycombinator.com/item?id=49563766", "by": "eloisant", "date": "2026-09-04", "stance": "recommend", "quote": "I recommend llama.cpp over ollama.", "excerpt": "I recommend llama.cpp over ollama. Anyway, when you see a model on huggingface you get the exact commands to copy paste to run it locally.", "context": "", "use_case": "running local models", "switched_from": "", "story": "Which tools do Claude, Codex and Cursor choose? We measured 17k runs to find out"}, {"id": 793011681, "item_id": 49563230, "url": "https://news.ycombinator.com/item?id=49563230", "by": "KronisLV", "date": "2026-09-04", "stance": "recommend", "quote": "there's llama.cpp and vLLM, each harder to setup but opens the door for squeezing more performance out of your hardware", "excerpt": "> Download ollama and give it a try Weren't people pissed off at them for a variety of reasons? Not giving attribution etc. In case anyone wants other good alternatives, there's llama.cpp and vLLM, each harder to setup but opens the door for squeezing more performance out of your hardware. I've also heard okay things about LM Studio and I think Unsloth had their own thing as well: https://unsloth.ai/docs/new/studio >", "context": "", "use_case": "running local LLMs", "switched_from": "", "story": "Which tools do Claude, Codex and Cursor choose? We measured 17k runs to find out"}, {"id": 792541393, "item_id": 49533837, "url": "https://news.ycombinator.com/item?id=49533837", "by": "jwr", "date": "2026-09-02", "stance": "recommend", "quote": "develop a local runner for `llama.cpp`", "excerpt": "For what it's worth, I've spent some time with Claude to develop a local runner for `llama.cpp`. I run Qwen3.6-35B-A3B-MTP (fast!) and Qwen3.8-27B (20 tokens/s). This was definitely worth the effort. Benchmarking and testing various approaches and various options really paid off. For example, one thing that surprised me was that MTP made things slower, not faster for Qwen3.8-27B. I use a 64GB MacBook Pro (M4 Max).", "context": "64GB MacBook Pro M4 Max user", "use_case": "running local models", "switched_from": "", "story": "Show HN: Running 104GB Qwen3.8-Flash-Next on 48GB Mac with at ~12 tok/s"}, {"id": 792225972, "item_id": 49514123, "url": "https://news.ycombinator.com/item?id=49514123", "by": "SamInTheShell", "date": "2026-08-31", "stance": "recommend", "quote": "handle running llama-server for me from any device that has the llama-cpp stack setup", "excerpt": "My use-case is only coding, every model sucks at writing good literature and there is no way around that (have had people try to debate me on this, but it's a taste thing, I have extensive English writing skills from my school years). Prior to two weeks ago, I was just using Pi and Ollama. I have tried my hand at putting together a few harnesses and I finally landed on what I like. Been working on this small app to h", "context": "", "use_case": "running local models", "switched_from": "Ollama", "story": "Apple caught off guard by AI demand for Mac Mini and Mac Studio"}, {"id": 792161106, "item_id": 49510069, "url": "https://news.ycombinator.com/item?id=49510069", "by": "inventor7777", "date": "2026-08-31", "stance": "neutral", "quote": "I use Qwen/GPT-OSS on my Mac Studio via llama.cpp server.", "excerpt": "One use case I find cost effective is using it as a voice assistant for Home Assistant. API pricing on models is very weird compared to the normal chats, so I use Qwen/GPT-OSS on my Mac Studio via llama.cpp server.", "context": "", "use_case": "serving local models", "switched_from": "", "story": "Apple caught off guard by AI demand for Mac Mini and Mac Studio"}, {"id": 791773008, "item_id": 49485813, "url": "https://news.ycombinator.com/item?id=49485813", "by": "wolvoleo", "date": "2026-08-29", "stance": "recommend", "quote": "I use Llama.cpp and OpenAI-style API of course", "excerpt": "I use Llama.cpp and OpenAI-style API of course (who doesn't, unless you use Ollama perhaps?) Ollama sucks, they dropped ROCm support for my card out of the blue, and the Vulkan replacement wasn't ready. The developer just shrugged and didn't care about the people affected so I moved to llama-server. Also, here's some other reasons to avoid ollama: https://sleepingrobots.com/dreams/stop-using-ollama/ And really, ollam", "context": "runs Linux server with multiple GPUs", "use_case": "running local LLMs", "switched_from": "Ollama", "story": "Run Qwen3.8 27B locally: real numbers from my Mac Studio"}, {"id": 791684160, "item_id": 49480260, "url": "https://news.ycombinator.com/item?id=49480260", "by": "cjdell", "date": "2026-08-28", "stance": "neutral", "quote": "Qwen 3.8 27B Q5_X_M on llama.cpp (Vulkan) with 256K context size", "excerpt": "An experiment to see what can be achieved using a single Radeon AI Pro R9700 32GB with Qwen 3.8 27B Q5_X_M on llama.cpp (Vulkan) with 256K context size. Total agentic crunching time about 24 hours with ~10 prompts along the way. Source: https://github.com/cjdell/qwencraft Live Dashboard: https://qwencraft.home.chrisdell.info/dashboard/", "context": "", "use_case": "running local model for agentic coding", "switched_from": "", "story": "Qwen 3.8 on 1x32GB Radeon wrote Minecraft clone in 24 hours (Rust, WebGPU, WASM)"}, {"id": 791266592, "item_id": 49454162, "url": "https://news.ycombinator.com/item?id=49454162", "by": "xlayn", "date": "2026-08-26", "stance": "neutral", "quote": "I merged llama.cpp tentative branches to get it running here", "excerpt": "For the impatient, I merged llama.cpp tentative branches to get it running here https://github.com/alainnothere/llama.cpp/tree/disk-cache-eviction, thing runs at 23.54 token/sec and my setup runs at high 30 the 3.8 dense 27B. and this is the pelican from the iq4_xs model https://github.com/alainnothere/llama.cpp/blob/disk-cache-eviction/media/pelicanRidingBycicle.jpg", "context": "", "use_case": "running Qwen locally", "switched_from": "", "story": "Qwen3.8-Flash-Next"}, {"id": 791184882, "item_id": 49449055, "url": "https://news.ycombinator.com/item?id=49449055", "by": "cyanydeez", "date": "2026-08-26", "stance": "recommend", "quote": "I use llamacpp with --reasoning-budget; very simple way to move on.", "excerpt": "My stack is basically deer-flow with Qwen3.5-122B-A10B; this hopefully will be a speed and intelligence improvement. Running deer-flow overnight on any research topic or verify clear scoped programming issue is really neat. Also, heating my home during the winter is nice. Oh, also, I use llamacpp with --reasoning-budget; very simple way to move on.", "context": "homelab user", "use_case": "running local models", "switched_from": "", "story": "Qwen3.8-Flash-Next"}, {"id": 790997568, "item_id": 49437348, "url": "https://news.ycombinator.com/item?id=49437348", "by": "SwellJoe", "date": "2026-08-25", "stance": "neutral", "quote": "Just a few GB for the OS and llama.cpp.", "excerpt": "No. In Linux, you can ignore the BIOS (well, set it to 512MB) and set GTT to allow almost the entirety of memory for GPU. If you don't run a desktop GUI (and why would you), you don't need memory for much of anything else. Just a few GB for the OS and llama.cpp. These are roughly the settings I use: https://github.com/kyuz0/amd-strix-halo-toolboxes#kernel-parameters-tested-on-fedora-42", "context": "runs local LLMs on AMD Strix Halo", "use_case": "running local LLMs", "switched_from": "", "story": "Qwen 3.8-Flash-Next releasing tomorrow (125B a6B)"}, {"id": 790955504, "item_id": 49434719, "url": "https://news.ycombinator.com/item?id=49434719", "by": "jlokier", "date": "2026-08-25", "stance": "recommend", "quote": "now use llama.cpp instead of Ollama", "excerpt": "You can't run Linux directly on these. Asahi Linux supports up to M2 only. Linux runs very well in a VM on macOS. There are many good options for this, some free and open source (QEMU, UTM, Lima, Colima), some proprietary (VMware Fusion, Parallels). But Linux in a VM doesn't get access to the real GPU, so model performance is limited. Those running on the CPU perform well, and those needing the GPU don't. However, ma", "context": "", "use_case": "local model runner on Mac", "switched_from": "Ollama", "story": "Apple introduces M6 and M5 Ultra"}, {"id": 790471296, "item_id": 49404456, "url": "https://news.ycombinator.com/item?id=49404456", "by": "zepearl", "date": "2026-08-22", "stance": "recommend", "quote": "I now absolutely love all the knobs that \"llama.cpp\" has to tune your hardware setup & your workload", "excerpt": "I know nothing about \"vLLM\" -> anything better compared to \"llama.cpp\"? I started with \"Ollama\" (precompiled version) and it worked and was good enough to understand the very basics. Then I downloaded the sourcecode of \"llama.cpp\", compiled it with specific compilation options for my GPUs (CUDA/nVidia using proprietary module on Gentoo Linux) & CPU (AMD), and the same model ran twice as fast -> since then I stuck wit", "context": "Gentoo Linux user with NVIDIA GPUs and AMD CPU", "use_case": "running local LLMs", "switched_from": "Ollama", "story": "Why your local LLM feels dumber than it is"}, {"id": 788644961, "item_id": 49290310, "url": "https://news.ycombinator.com/item?id=49290310", "by": "SwellJoe", "date": "2026-08-13", "stance": "recommend", "quote": "they currently work great with llama.cpp and current ROCm 7.14", "excerpt": "\"used to\" in your sentence already answers your curiosity. ROCm runs most things just fine without any ceremony or difficulty. I have a couple of the same cards as covered in the article, bought before they got expensive (I'd recommend current Radeon AI Pro 9700 cards over the V620 now, though, as they have increased in price since I bought mine), and they're at the long end of the supported chart for ROCm...nearly E", "context": "", "use_case": "local inference", "switched_from": "", "story": "We Have AI at Home Chapter 1: A Box of Scraps"}, {"id": 788296288, "item_id": 49268518, "url": "https://news.ycombinator.com/item?id=49268518", "by": "imrehg", "date": "2026-08-11", "stance": "recommend", "quote": "I guess overall it's the worst runtime I've seen so far, except for all the other runtimes out there... I'm a fan", "excerpt": "llama.cpp works pretty well for me on the Framework 13 laptop, but the current era of \"move fast, break things, rarely fix\" (sorry, that's how it feels), bites here quite a bit. Two examples: - https://github.com/ggml-org/llama.cpp/pull/25863 Someone's few lines change broke the native (ROCm) support for the AMD GPU inside Framework (and other integrated systems), and any rollback or proper fix is pending for almost ", "context": "Framework 13 laptop user with AMD GPU", "use_case": "local LLM inference on Framework 13", "switched_from": "", "story": "llama.cpp"}, {"id": 784331137, "item_id": 49020696, "url": "https://news.ycombinator.com/item?id=49020696", "by": "gbuk2013", "date": "2026-07-23", "stance": "recommend", "quote": "there was some fiddling required to get it to use GPU (I use llama.cpp)", "excerpt": "I run Qwen3-Coder-Next-UD-Q4_K_XL and other than the initial wait to initialise context (which takes less than 2 minutes) subsequent prompts return in less than a minute, usually less than 30s. If your performance is significantly slower then you are probably doing it in CPU - there was some fiddling required to get it to use GPU (I use llama.cpp)", "context": "", "use_case": "local LLM inference", "switched_from": "", "story": "New Framework Desktop Option with AMD Ryzen AI Max+ Pro 495 and 192GB Memory"}, {"id": 783500929, "item_id": 48968808, "url": "https://news.ycombinator.com/item?id=48968808", "by": "ollamer", "date": "2026-07-19", "stance": "recommend", "quote": "On llama.cpp I can get models like Qwen 35BA3B running partially on gpu/cpu with 40t/s on a laptop thanks to --n-cpu-moe", "excerpt": "https://github.com/ollama/ollama/issues/11772 A year and still no implementation for such a basic need as offloading MoE layers onto the CPU selectively. On llama.cpp I can get models like Qwen 35BA3B running partially on gpu/cpu with 40t/s on a laptop thanks to --n-cpu-moe but on this VC funded joke it would be simply unusable. I can't quite understand how you make a wrapper so much worse than the code you're rippin", "context": "", "use_case": "partial CPU/GPU MoE inference", "switched_from": "", "story": "Ollama: All Aboard Open Models"}, {"id": 782202704, "item_id": 48887669, "url": "https://news.ycombinator.com/item?id=48887669", "by": "MyMemoryfails", "date": "2026-07-12", "stance": "recommend", "quote": "Running Local LLM isn't really a lot effort anymore, since its one command away", "excerpt": "Running Local LLM isn't really a lot effort anymore, since its one command away to get the familar interface of chat AI. This setup requires 24GB ish VRAM and 32GB ram, so if you have capable gaming PC which can do AAA games, you can run it. - llama-server -hf ggml-org/gemma-4-26b-a4b-it-GGUF:Q4_K_M After that simply open browser and enter: http://localhost:8080 What this does: This will download Gemma4 AI with 26B p", "context": "", "use_case": "running local LLMs", "switched_from": "", "story": "Ask HN: How do you use LLMs for private discussions?"}, {"id": 776749920, "item_id": 48546870, "url": "https://news.ycombinator.com/item?id=48546870", "by": "jboss10", "date": "2026-06-15", "stance": "recommend", "quote": "llama.cpp It's faster and more open source.", "excerpt": "llama.cpp It's faster and more open source. Ollama has some mixed history. I use llama-swap to emulate the Ollama experience.", "context": "", "use_case": "local LLM inference", "switched_from": "", "story": "Ask HN: Has anyone replaced Claude/GPT with a local model for daily coding?"}, {"id": 776719776, "item_id": 48544986, "url": "https://news.ycombinator.com/item?id=48544986", "by": "porkloin", "date": "2026-06-15", "stance": "recommend", "quote": "llama.cpp vulkan (better performance than ROCM, though I'm keeping an eye on it in the future)", "excerpt": "I have good results with this setup: Hardware: - GPU: AMD 7900xtx, 24gb vram - CPU: AMD 5950x, AM4 - RAM: 64gb DDR4 3600 Software: - OS: Bazzite (atomic fedora - this machine is running Steam \"big picture\" mode on my TV when not in use for LLM tasks) - Virtualization: Podman Quadlets, which allows me to run container images as managed systemd units - Network: tailscale - Inference: llama.cpp vulkan (better performanc", "context": "", "use_case": "local LLM inference", "switched_from": "", "story": "Ask HN: Has anyone replaced Claude/GPT with a local model for daily coding?"}, {"id": 776006098, "item_id": 48500381, "url": "https://news.ycombinator.com/item?id=48500381", "by": "dlcarrier", "date": "2026-06-11", "stance": "warn", "quote": "I can't get Llama.cpp to use the GPU", "excerpt": "I have an Intel A770, because at $200 USD it was the second cheapest way to get 16 of dedicated VRAM. (The cheapest is a $100 Radeon Instinct Mi25, but it's a bit old and slow.) I battled with Python packages a lot to get a1111 working, gave up when it broke on an update and switched to ComfyUI which has worked great. I can't get Llama.cpp to use the GPU, but I'm currently installing some of Intel's proprietary OpenV", "context": "", "use_case": "LLM on Intel GPU", "switched_from": "", "story": "Ask HN: What computer are you using for AI coding tools?"}]}