llama.cpp
Open-source C/C++ engine for running LLM inference locally on CPUs and GPUs.
Recommended over Ollama as a faster, more tunable way to run local models1234, with a built-in server, web UI and broad GPU backend support.5678 The most common caveat is GPU setup friction91011, with ROCm crashes and regressions often pushing users to the Vulkan build.121314
Why people recommend it
Ollama alternative · 3
Many switched from Ollama, citing better speed, features and openness.
“I now absolutely love all the knobs that "llama.cpp" has to tune your hardware setup & your workload”
“I use Llama.cpp and OpenAI-style API of course”
Show 1 more
“On llama.cpp I can get models like Qwen 35BA3B running partially on gpu/cpu with 40t/s on a laptop thanks to --n-cpu-moe”
Performance tuning · 3
Lots of knobs and local builds let users get much more speed from their hardware.
“I now absolutely love all the knobs that "llama.cpp" has to tune your hardware setup & your workload”
“found a lot of performance improvement over ollama by compiling llama.cpp and running with —no-mmap and —perf”
Show 1 more
“but for how we use it llama.cpp is still better for us.”
Rich features · 3
Commenters cite reasoning budgets, grammars, structured outputs, memory mapping and benchmarking tools.
“--thinking-budget and --thinking-message is all you need in llamacpp to keep it progressing.”
“llama.cpp supports structured outputs, as do some more consumer-oriented tools that wrap like LM Studio.”
Show 1 more
“llama.cpp includes a tool specifically for benchmarking that will sweep the arguments for you”
Server and web UI · 3
llama-server provides a built-in web UI, OpenAI-style API and multi-model routing.
“llama.cpp is simply better with its awesome built-in webui, router and server APIs”
“llama-server's supported multiple models for a while now”
Show 1 more
“Running Local LLM isn't really a lot effort anymore, since its one command away”
Hardware support · 3
Works across Vulkan, ROCm, CUDA, SYCL and Apple hardware, including pooling VRAM across cards.
“Using llama.cpp Vulkan let's you pool VRAM across cards”
“they currently work great with llama.cpp and current ROCm 7.14”
Show 1 more
“llama.cpp is pretty exciting right now with the SYCL improvements”
Active community · 3
Maintainers and contributors add new models and fixes quickly.
“I use either llama.cpp itself or projects that depend on it all the time”
“I highly recommend using llama.cpp for running AI model locally vs using other inference framework”
Show 1 more
“It does seem like llama.cpp is successfully riding that dragon right now though.”
Caveats and dissent
GPU setup · 2
Getting the GPU used can take fiddling, and one user couldn't get it working on an Intel Arc card.
“I can't get Llama.cpp to use the GPU”
“there was some fiddling required to get it to use GPU (I use llama.cpp)”
ROCm breakage · 2
Users report ROCm crashes and regressions that remain unfixed, with Vulkan as the workaround.
“I guess overall it's the worst runtime I've seen so far, except for all the other runtimes out there... I'm a fan”
“so I switched to the Windows Vulkan build of llama.cpp instead, which worked reliably.”
Fast-moving releases · 2
Users rebuild often to keep up, and changes sometimes break things.
“I recommend building recent llama.cpp from source, I've been updating about once a week”
“I guess overall it's the worst runtime I've seen so far, except for all the other runtimes out there... I'm a fan”
Harder setup · 1
Harder to set up than simpler wrappers, though more performant.
“there's llama.cpp and vLLM, each harder to setup but opens the door for squeezing more performance out of your hardware”
Often compared with
14 moved from Ollama to llama.cpp
2 moved from vLLM to llama.cpp
1 moved from llama.cpp to vLLM
1 moved from LM Studio to llama.cpp
1 moved from llama.cpp to LM Studio
Who recommends it in their own words
From the makers not counted in the ranking
“2 CUDA PRs merged into llama.cpp”
“I keep this branch of llama.cpp that includes that same functionality”
“you have to use my version of llama.cpp in the link to try that”
“It's been tested 99% of the time against a local llama.cpp”
“We are preparing official semantic versioning of llama.cpp”
“I forked llama.cpp and added some crude mechanism to keep exactly this behavior under control”
Threads where it came up
- Ask HN: Has anyone replaced Claude/GPT with a local model for daily coding? 16 mentions
- llama.cpp 9 mentions
- Running local models is good now 9 mentions
- Qwen 3.8 27B is out: open weights, best local dense model yet 5 mentions
- Qwen 3.8 27B is excellent, but it defaults to overthinking things 4 mentions
- Ask HN: Who uses open LLMs and coding assistants locally? Share setup and laptop 3 mentions
Also written as Llama.cpp. Data: llama-cpp.json. Quotes are verbatim; the verdict and theme labels are written by a model from these comments, and every clause links to its sources.
All comments 49
“32 parallel sequences for your llamacpp (or similar)”
“many Qwen-derived models run fine without any changes to llama.cpp”
“I tried llama.cpp with offloading "all what could be" to my GPU”
“it works fine with offloading to GPU and using llama.cpp”
“llama-server's supported multiple models for a while now”
“llama.cpp serving gemma4-26b-a4b for image processing of things too sensitive to let openai and other similar companies to see it”
“these days I've figure out llama.cpp and proper context sizes”
“one tab is GitBash running llama.ccp and the other is normal terminal running pi. Winning combo for me.”
“Since it seems llama.cpp did not optimize that path, we used 16-bit KV instead.”
“Then it rebuilds latest llama.cpp, grabs the PRs it finds relevant to test against”
“Using llama.cpp Vulkan let's you pool VRAM across cards”
“I use this solution for your exact use case”
“Lemire absolutely crushed it! I benchmarked his change today.”
“That's what I've done when playing around with llama.”
“llama.cpp + llama-swap”
“llama.cpp has a web ui nowdays”
“I use a custom fork of llama.cpp that has an abliteration feature”
“you need Prism's llama.cpp fork to get them to work”
“llama.cpp already makes recommendations on what models to run.”
“later grew support for continuing conversations through llama.cpp or OpenRouter”
“More ethical alternatives have caught up including llama.cpp”
“served by llama.cpp, Freetoken, or vLLM”
“done through llama.cpp's web UI”
“llama.cpp is simply better with its awesome built-in webui, router and server APIs”
“llama.cpp master is still somewhat bad in Qwen 3.8 next performance”
“so I switched to the Windows Vulkan build of llama.cpp instead, which worked reliably.”
“I keep ending up with the same problem when running llama.cpp”
“It does seem like llama.cpp is successfully riding that dragon right now though.”
“I use either llama.cpp itself or projects that depend on it all the time”
“I recommend llama.cpp over ollama.”
“there's llama.cpp and vLLM, each harder to setup but opens the door for squeezing more performance out of your hardware”
“develop a local runner for `llama.cpp`”
“handle running llama-server for me from any device that has the llama-cpp stack setup”
“I use Qwen/GPT-OSS on my Mac Studio via llama.cpp server.”
“I use Llama.cpp and OpenAI-style API of course”
“Qwen 3.8 27B Q5_X_M on llama.cpp (Vulkan) with 256K context size”
“I merged llama.cpp tentative branches to get it running here”
“I use llamacpp with --reasoning-budget; very simple way to move on.”
“Just a few GB for the OS and llama.cpp.”
“now use llama.cpp instead of Ollama”
“I now absolutely love all the knobs that "llama.cpp" has to tune your hardware setup & your workload”
“they currently work great with llama.cpp and current ROCm 7.14”
“I guess overall it's the worst runtime I've seen so far, except for all the other runtimes out there... I'm a fan”
“there was some fiddling required to get it to use GPU (I use llama.cpp)”
“On llama.cpp I can get models like Qwen 35BA3B running partially on gpu/cpu with 40t/s on a laptop thanks to --n-cpu-moe”
“Running Local LLM isn't really a lot effort anymore, since its one command away”
“llama.cpp It's faster and more open source.”
“llama.cpp vulkan (better performance than ROCM, though I'm keeping an eye on it in the future)”
“I can't get Llama.cpp to use the GPU”