{"slug": "unsloth", "name": "Unsloth", "category": "ai", "category_name": "AI models and assistants", "description": "Open-source toolkit for fast LLM fine-tuning that also publishes quantized GGUF model weights.", "recommenders": 9, "commenters": 18, "warners": 3, "score": 7.003, "rising": -0.62, "first": "2026-02-27", "latest": "2026-09-11", "why": "Recommended mainly for its quantized GGUF models, which commenters say hold up well at lower bit widths, and for running and fine-tuning your own local LLMs.", "why_is_quote": false, "has_verdict": true, "sentiment": {"recommend": 10, "neutral": 7, "compare": 0, "warn": 3}, "spark": [0, 0, 0, 2, 1, 3, 0, 1, 1, 1, 1, 0], "listed": true, "loved": ["Quant quality", "Fixes issues"], "caveat": "Context room", "short_quote": "Unsloth is the real thing. Highly recommended for those running their own AI engines", "icon": true, "domain": null, "github_repo": "github.com/sponsors/unslothai", "official_url": "https://unsloth.ai", "pricing": "open source", "aliases": ["Unsloth"], "verdict": [{"text": "Recommended mainly for its quantized GGUF models, which commenters say hold up well at lower bit widths", "cites": [{"id": 780519682, "item_id": 48782480, "url": "https://news.ycombinator.com/item?id=48782480", "by": "SwellJoe", "date": "2026-07-03", "stance": "recommend", "quote": "if using dynamic quantizations like unsloth provides", "excerpt": "The Gemma 4 QAT 4-bit models are basically indistinguishable from the 8-bit or 16-bit versions. I tested, and many benchmarks agree. For models where there isn't a QAT version, 6-bit is where it starts to degrade, but only a little if using dynamic quantizations like unsloth provides. When I use Qwen, I use the 8-bit quantization. But, when I use Gemma there is literally no reason to use anything other than the 4-bit", "context": "", "use_case": "dynamic quantization", "switched_from": "", "story": "Kimi K2.7 Code is generally available in GitHub Copilot"}, {"id": 776919472, "item_id": 48557467, "url": "https://news.ycombinator.com/item?id=48557467", "by": "c0rruptbytes", "date": "2026-06-16", "stance": "recommend", "quote": "I recommend unsloth quants, i recommend 6bit for MoEs and 5bit for dense", "excerpt": "I don't know about good, I use a lot of local models and they're still pretty painful to run locally You have dense models (qwen 27b, gemma 31b) who are pretty smart, but pretty slow You have MoE models (gemma 26b, qwen 35b, north mini code 30b) who are pretty fast, but make a lot of mistakes You need a lot of memory to run these well, quantization makes tool calling weaker, so most run at 4 bit quants and are wonder", "context": "", "use_case": "quantized local models", "switched_from": "", "story": "Running local models is good now"}, {"id": 765996864, "item_id": 47874804, "url": "https://news.ycombinator.com/item?id=47874804", "by": "syntaxing", "date": "2026-04-23", "stance": "recommend", "quote": "I had similar issue until I used Unsloth\u2019s.", "excerpt": "Which quants are you using? I had similar issue until I used Unsloth\u2019s. I would recommend at least UD_6. Also, make sure your context length is above 65K. https://huggingface.co/unsloth/Qwen3.6-35B-A3B-GGUF", "context": "", "use_case": "quantized local Qwen models", "switched_from": "", "story": "Qwen3.6-27B: Flagship-Level Coding in a 27B Dense Model"}, {"id": 788508608, "item_id": 49281788, "url": "https://news.ycombinator.com/item?id=49281788", "by": "SwellJoe", "date": "2026-08-12", "stance": "recommend", "quote": "almost always, I pick the unsloth quantization if there is one", "excerpt": "Counterpoint: I've been using the Unsloth Gemma 4 quants extensively since very soon after release (on ROCm and Apple Silicon, I don't have any Nvidia hardware big enough), pretty much every quantization down to 4 bits (the QAT is the business, indistinguishable from the full-fat version, runs great on a slightly chonky desktop or laptop), and I haven't had any issues. The reason I use Gemma 4 so much often comes dow", "context": "Runs local models on ROCm and Apple Silicon", "use_case": "quantized local models", "switched_from": "", "story": "Qwen/Qwen3.8-2.4T-A95B"}]}, {"text": ", and for running and fine-tuning your own local LLMs.", "cites": [{"id": 758733920, "item_id": 47420870, "url": "https://news.ycombinator.com/item?id=47420870", "by": "jacquesm", "date": "2026-03-17", "stance": "recommend", "quote": "Unsloth is the real thing. Highly recommended for those running their own AI engines", "excerpt": "Unsloth is the real thing. Highly recommended for those running their own AI engines and that want to get the most out of them.", "context": "", "use_case": "running local AI models", "switched_from": "", "story": "Unsloth Studio"}, {"id": 755067298, "item_id": 47191706, "url": "https://news.ycombinator.com/item?id=47191706", "by": "segmondy", "date": "2026-02-27", "stance": "recommend", "quote": "the unsloth quants are great", "excerpt": "This is a good time to promote running your own models. I have been running my own models locally and I would wager a local model will meet 85-95% of your needs if you really learn to use it. These models have gotten great. For anyone wanting to get into this, the smartest models to run recently that is consumer friendly was just released, checkout Qwen3.5 the 27B and 35B variants. They are small and I recommend runn", "context": "runs models locally", "use_case": "quantized model weights", "switched_from": "", "story": "How do I cancel my ChatGPT subscription?"}, {"id": 766259153, "item_id": 47891197, "url": "https://news.ycombinator.com/item?id=47891197", "by": "mchusma", "date": "2026-04-24", "stance": "recommend", "quote": "Run through unsloth, and with some fine tuning", "excerpt": "At first, I was more excited about the Flash model, but I'm now more excited about the Pro model in many ways. I feel like the Pro model with an Run through unsloth, and with some fine tuning, is gonna be enough for many vertical SaaS applications. Where previously I was wary to under-provide the intelligence level, I'm now more excited about the idea of being able to give these pretty large intelligent models to my ", "context": "builds vertical SaaS applications", "use_case": "fine-tuning models", "switched_from": "", "story": "DeepSeek v4"}]}, {"text": " Complaints are scattered rather than shared: one user got far less context room than with regular quants", "cites": [{"id": 789912272, "item_id": 49369517, "url": "https://news.ycombinator.com/item?id=49369517", "by": "zenoprax", "date": "2026-08-19", "stance": "warn", "quote": "For some reason the unsloth models leave hardly any room for context.", "excerpt": "For some reason the unsloth models leave hardly any room for context. I've switched to the regular (non-unsloth) and get about 25 t/s and get about 80,000 more context tokens for the same quant.", "context": "", "use_case": "local quantized models", "switched_from": "", "story": "Unsloth Dynamic 3.0 GGUFs"}]}, {"text": ", and others flag hacky llama.cpp support for one model and weak vLLM support for GGUF.", "cites": [{"id": 778079521, "item_id": 48629970, "url": "https://news.ycombinator.com/item?id=48629970", "by": "xrd", "date": "2026-06-22", "stance": "warn", "quote": "it hacks a bunch of things to make it work and isn't optimal", "excerpt": "How are people running this locally? I just checked llama.cpp and it appears unsloth has a version but it hacks a bunch of things to make it work and isn't optimal. https://github.com/ggml-org/llama.cpp/issues/24730", "context": "", "use_case": "local model version", "switched_from": "", "story": "GLM 5.2 vs. Opus"}, {"id": 755129488, "item_id": 47195593, "url": "https://news.ycombinator.com/item?id=47195593", "by": "deepsquirrelnet", "date": "2026-02-28", "stance": "recommend", "quote": "I love the work unsloth is doing.", "excerpt": "I love the work unsloth is doing. I only wish gguf format had better vllm support. It\u2019s sometimes hard to find trustworthy quants that work well with vllm.", "context": "", "use_case": "quantized models", "switched_from": "", "story": "Unsloth Dynamic 2.0 GGUFs"}]}], "praise": [{"label": "Quant quality", "summary": "Commenters say Unsloth's dynamic quants degrade less and are their default choice when available.", "quotes": [{"id": 788508608, "item_id": 49281788, "url": "https://news.ycombinator.com/item?id=49281788", "by": "SwellJoe", "date": "2026-08-12", "stance": "recommend", "quote": "almost always, I pick the unsloth quantization if there is one", "excerpt": "Counterpoint: I've been using the Unsloth Gemma 4 quants extensively since very soon after release (on ROCm and Apple Silicon, I don't have any Nvidia hardware big enough), pretty much every quantization down to 4 bits (the QAT is the business, indistinguishable from the full-fat version, runs great on a slightly chonky desktop or laptop), and I haven't had any issues. The reason I use Gemma 4 so much often comes dow", "context": "Runs local models on ROCm and Apple Silicon", "use_case": "quantized local models", "switched_from": "", "story": "Qwen/Qwen3.8-2.4T-A95B"}, {"id": 780519682, "item_id": 48782480, "url": "https://news.ycombinator.com/item?id=48782480", "by": "SwellJoe", "date": "2026-07-03", "stance": "recommend", "quote": "if using dynamic quantizations like unsloth provides", "excerpt": "The Gemma 4 QAT 4-bit models are basically indistinguishable from the 8-bit or 16-bit versions. I tested, and many benchmarks agree. For models where there isn't a QAT version, 6-bit is where it starts to degrade, but only a little if using dynamic quantizations like unsloth provides. When I use Qwen, I use the 8-bit quantization. But, when I use Gemma there is literally no reason to use anything other than the 4-bit", "context": "", "use_case": "dynamic quantization", "switched_from": "", "story": "Kimi K2.7 Code is generally available in GitHub Copilot"}, {"id": 776919472, "item_id": 48557467, "url": "https://news.ycombinator.com/item?id=48557467", "by": "c0rruptbytes", "date": "2026-06-16", "stance": "recommend", "quote": "I recommend unsloth quants, i recommend 6bit for MoEs and 5bit for dense", "excerpt": "I don't know about good, I use a lot of local models and they're still pretty painful to run locally You have dense models (qwen 27b, gemma 31b) who are pretty smart, but pretty slow You have MoE models (gemma 26b, qwen 35b, north mini code 30b) who are pretty fast, but make a lot of mistakes You need a lot of memory to run these well, quantization makes tool calling weaker, so most run at 4 bit quants and are wonder", "context": "", "use_case": "quantized local models", "switched_from": "", "story": "Running local models is good now"}]}, {"label": "Fixes issues", "summary": "Users report that switching to Unsloth quants resolved problems they had with other quants.", "quotes": [{"id": 765996864, "item_id": 47874804, "url": "https://news.ycombinator.com/item?id=47874804", "by": "syntaxing", "date": "2026-04-23", "stance": "recommend", "quote": "I had similar issue until I used Unsloth\u2019s.", "excerpt": "Which quants are you using? I had similar issue until I used Unsloth\u2019s. I would recommend at least UD_6. Also, make sure your context length is above 65K. https://huggingface.co/unsloth/Qwen3.6-35B-A3B-GGUF", "context": "", "use_case": "quantized local Qwen models", "switched_from": "", "story": "Qwen3.6-27B: Flagship-Level Coding in a 27B Dense Model"}, {"id": 779619201, "item_id": 48726200, "url": "https://news.ycombinator.com/item?id=48726200", "by": "anon373839", "date": "2026-06-29", "stance": "neutral", "quote": "I\u2019m using Unsloth\u2019s Q8_K_XL quant", "excerpt": "I use Qwen 3.6 35B-A3B constantly, but I don\u2019t see the type of behavior you mentioned. I\u2019m using Unsloth\u2019s Q8_K_XL quant.", "context": "", "use_case": "quantized model weights", "switched_from": "", "story": "Qwen 3.6 27B is the sweet spot for local development"}]}, {"label": "Consumer hardware", "summary": "Users run Unsloth quants with long context on Macs and single consumer GPUs.", "quotes": [{"id": 792240321, "item_id": 49515020, "url": "https://news.ycombinator.com/item?id=49515020", "by": "jckahn", "date": "2026-08-31", "stance": "neutral", "quote": "I use Unsloth's Q3 XXS with 128k context.", "excerpt": "I run Qwen 3.8 27B just fine on my Mac mini M4 24GB. I use Unsloth's Q3 XXS with 128k context. It successfully completes long horizon tasks with OpenCode.", "context": "", "use_case": "quantized local model", "switched_from": "", "story": "Apple caught off guard by AI demand for Mac Mini and Mac Studio"}, {"id": 779606529, "item_id": 48725408, "url": "https://news.ycombinator.com/item?id=48725408", "by": "cpburns2009", "date": "2026-06-29", "stance": "neutral", "quote": "I use unsloth's UD-Q5_K_XL at 256k context", "excerpt": "A 32gb card does run it nicely. I use unsloth's UD-Q5_K_XL at 256k context (k/v at q8_0), and get ~67 t/s on a 5090. I still need to look into MTP.", "context": "", "use_case": "quantized model weights", "switched_from": "", "story": "Qwen 3.6 27B is the sweet spot for local development"}, {"id": 788508608, "item_id": 49281788, "url": "https://news.ycombinator.com/item?id=49281788", "by": "SwellJoe", "date": "2026-08-12", "stance": "recommend", "quote": "almost always, I pick the unsloth quantization if there is one", "excerpt": "Counterpoint: I've been using the Unsloth Gemma 4 quants extensively since very soon after release (on ROCm and Apple Silicon, I don't have any Nvidia hardware big enough), pretty much every quantization down to 4 bits (the QAT is the business, indistinguishable from the full-fat version, runs great on a slightly chonky desktop or laptop), and I haven't had any issues. The reason I use Gemma 4 so much often comes dow", "context": "Runs local models on ROCm and Apple Silicon", "use_case": "quantized local models", "switched_from": "", "story": "Qwen/Qwen3.8-2.4T-A95B"}]}, {"label": "Recommended settings", "summary": "Unsloth's recommended sampler and temperature settings are cited as worth following.", "quotes": [{"id": 766822003, "item_id": 47926375, "url": "https://news.ycombinator.com/item?id=47926375", "by": "jonaustin", "date": "2026-04-27", "stance": "recommend", "quote": "make sure you're following Unsloth's recommendations for temperature/etc.", "excerpt": "Try https://github.com/jundot/omlx I've been using qwen3.5 (122b) with claude code for months, and it's definitely dumber than sonnet/opus, but it works through things reasonably well (i.e. writes half-decent code and tool calls usually work), and I pretty much never run into loops now. and make sure you're following Unsloth's recommendations for temperature/etc.", "context": "", "use_case": "local model settings", "switched_from": "", "story": "Running Local LLMs Offline on a Ten-Hour Flight"}, {"id": 794129634, "item_id": 49633102, "url": "https://news.ycombinator.com/item?id=49633102", "by": "spijdar", "date": "2026-09-09", "stance": "neutral", "quote": "This is Unsloth's UD-Q4_K_S quantization", "excerpt": "This is Unsloth's UD-Q4_K_S quantization (edit -- on llama.cpp, via the Vulkan backend, on an RX 7900 XT, with Unsloth's recommended sampler config), for \"as replicable as LLMs can be\" disclosure, done through llama.cpp's web UI. In another thread, via Open WebUI (both with no custom prompt, so whatever the \"default\" system prompt is), a reasoning block opens as such: We need answer user's request: write complete sho", "context": "", "use_case": "quantized model weights", "switched_from": "", "story": "Qwen 3.8 follows GPT-5.5 Pro reasoning prefills"}]}, {"label": "Worth funding", "summary": "Some see Unsloth as valuable open-source work deserving support.", "quotes": [{"id": 794592977, "item_id": 49662061, "url": "https://news.ycombinator.com/item?id=49662061", "by": "armchairhacker", "date": "2026-09-11", "stance": "recommend", "quote": "Unsloth (https://github.com/sponsors/unslothai)", "excerpt": "Even ignoring politics, I\u2019m sure there are more impactful ways to spend $18 million. Many OSS projects are underfunded, why not give some of that to e.g. Bevy (https://bevy.org/sponsorship-pledge/), Unsloth (https://github.com/sponsors/unslothai), or better, many small grants to many small projects?", "context": "", "use_case": "worthy OSS project to fund", "switched_from": "", "story": "Omarchy Is a Power Grab"}, {"id": 755129488, "item_id": 47195593, "url": "https://news.ycombinator.com/item?id=47195593", "by": "deepsquirrelnet", "date": "2026-02-28", "stance": "recommend", "quote": "I love the work unsloth is doing.", "excerpt": "I love the work unsloth is doing. I only wish gguf format had better vllm support. It\u2019s sometimes hard to find trustworthy quants that work well with vllm.", "context": "", "use_case": "quantized models", "switched_from": "", "story": "Unsloth Dynamic 2.0 GGUFs"}, {"id": 758733920, "item_id": 47420870, "url": "https://news.ycombinator.com/item?id=47420870", "by": "jacquesm", "date": "2026-03-17", "stance": "recommend", "quote": "Unsloth is the real thing. Highly recommended for those running their own AI engines", "excerpt": "Unsloth is the real thing. Highly recommended for those running their own AI engines and that want to get the most out of them.", "context": "", "use_case": "running local AI models", "switched_from": "", "story": "Unsloth Studio"}]}, {"label": "Fine-tuning", "summary": "One commenter plans to fine-tune models through Unsloth for vertical SaaS sub-agents.", "quotes": [{"id": 766259153, "item_id": 47891197, "url": "https://news.ycombinator.com/item?id=47891197", "by": "mchusma", "date": "2026-04-24", "stance": "recommend", "quote": "Run through unsloth, and with some fine tuning", "excerpt": "At first, I was more excited about the Flash model, but I'm now more excited about the Pro model in many ways. I feel like the Pro model with an Run through unsloth, and with some fine tuning, is gonna be enough for many vertical SaaS applications. Where previously I was wary to under-provide the intelligence level, I'm now more excited about the idea of being able to give these pretty large intelligent models to my ", "context": "builds vertical SaaS applications", "use_case": "fine-tuning models", "switched_from": "", "story": "DeepSeek v4"}]}], "caveats": [{"label": "Context room", "summary": "One user found Unsloth models left much less room for context than regular quants.", "quotes": [{"id": 789912272, "item_id": 49369517, "url": "https://news.ycombinator.com/item?id=49369517", "by": "zenoprax", "date": "2026-08-19", "stance": "warn", "quote": "For some reason the unsloth models leave hardly any room for context.", "excerpt": "For some reason the unsloth models leave hardly any room for context. I've switched to the regular (non-unsloth) and get about 25 t/s and get about 80,000 more context tokens for the same quant.", "context": "", "use_case": "local quantized models", "switched_from": "", "story": "Unsloth Dynamic 3.0 GGUFs"}]}, {"label": "llama.cpp hacks", "summary": "One commenter says Unsloth's llama.cpp version of a model relied on non-optimal hacks.", "quotes": [{"id": 778079521, "item_id": 48629970, "url": "https://news.ycombinator.com/item?id=48629970", "by": "xrd", "date": "2026-06-22", "stance": "warn", "quote": "it hacks a bunch of things to make it work and isn't optimal", "excerpt": "How are people running this locally? I just checked llama.cpp and it appears unsloth has a version but it hacks a bunch of things to make it work and isn't optimal. https://github.com/ggml-org/llama.cpp/issues/24730", "context": "", "use_case": "local model version", "switched_from": "", "story": "GLM 5.2 vs. Opus"}]}, {"label": "vLLM support", "summary": "GGUF quants have limited vLLM support, making working quants hard to find.", "quotes": [{"id": 755129488, "item_id": 47195593, "url": "https://news.ycombinator.com/item?id=47195593", "by": "deepsquirrelnet", "date": "2026-02-28", "stance": "recommend", "quote": "I love the work unsloth is doing.", "excerpt": "I love the work unsloth is doing. I only wish gguf format had better vllm support. It\u2019s sometimes hard to find trustworthy quants that work well with vllm.", "context": "", "use_case": "quantized models", "switched_from": "", "story": "Unsloth Dynamic 2.0 GGUFs"}]}, {"label": "Low-bit speed", "summary": "A 2-bit quant on a 32GB MacBook ran too slowly to be usable.", "quotes": [{"id": 793831312, "item_id": 49614457, "url": "https://news.ycombinator.com/item?id=49614457", "by": "mrbonner", "date": "2026-09-08", "stance": "neutral", "quote": "I use a 2-bit quant from Unsloth on my MBP M5 32GB of RAM.", "excerpt": "I use a 2-bit quant from Unsloth on my MBP M5 32GB of RAM. It run slower than molasses at 2 too/s kind of thing. Not sure it is usable at that rate for anything.", "context": "", "use_case": "running local quantized model", "switched_from": "", "story": "Benchmarking Qwen3.8 27B quantizations: 4-bit holds up, 1-bit collapses"}]}, {"label": "Community moderation", "summary": "One researcher reported a hostile, message-deleting exchange in Unsloth's community channel.", "quotes": [{"id": 777952176, "item_id": 48622011, "url": "https://news.ycombinator.com/item?id=48622011", "by": "verdverm", "date": "2026-06-21", "stance": "warn", "quote": "They remained accusatory, made it personal, and then started deleting messages.", "excerpt": "Sure, I shared a project in their #research channel where I used their qwen36moe quant to refresh my PhD research. The channel had a topic that ended with something like \"and all things research...\" One of their people accused me of self-promotion, and I reiterated that I shared it in that channel because it was their quant doing something (I thought) interesting as a research model. They remained accusatory, made it", "context": "PhD researcher", "use_case": "quantized models for research", "switched_from": "", "story": "Two Qwen3 models on one DGX Spark: the residency math"}]}], "who": [{"by": "SwellJoe", "context": "Runs local models on ROCm and Apple Silicon", "url": "https://news.ycombinator.com/item?id=49281788", "date": "2026-08-12"}, {"by": "mchusma", "context": "builds vertical SaaS applications", "url": "https://news.ycombinator.com/item?id=47891197", "date": "2026-04-24"}, {"by": "segmondy", "context": "runs models locally", "url": "https://news.ycombinator.com/item?id=47191706", "date": "2026-02-27"}], "tags": ["for local LLM users", "for Apple Silicon users", "for AMD GPU users", "for llama.cpp users", "for fine-tuning"], "use_cases": ["quantized model weights", "quantized local model", "quantized local models", "worthy oss project to fund", "running local quantized model", "local quantized models"], "compared": [{"slug": "llama-cpp", "name": "llama.cpp", "co": 5, "switched_from": 0, "switched_to": 0}, {"slug": "qwen-3-6", "name": "Qwen 3.6", "co": 5, "switched_from": 0, "switched_to": 0}, {"slug": "gemma", "name": "Gemma", "co": 3, "switched_from": 0, "switched_to": 0}, {"slug": "qwen-3-8", "name": "Qwen 3.8", "co": 2, "switched_from": 0, "switched_to": 0}, {"slug": "qwen", "name": "Qwen", "co": 2, "switched_from": 0, "switched_to": 0}, {"slug": "claude-opus", "name": "Claude Opus", "co": 2, "switched_from": 0, "switched_to": 0}, {"slug": "claude-sonnet", "name": "Claude Sonnet", "co": 2, "switched_from": 0, "switched_to": 0}, {"slug": "ollama", "name": "Ollama", "co": 2, "switched_from": 0, "switched_to": 0}], "makers": [], "threads": [{"url": "https://news.ycombinator.com/item?id=48721903", "title": "Qwen 3.6 27B is the sweet spot for local development", "mentions": 2}, {"url": "https://news.ycombinator.com/item?id=49658869", "title": "Omarchy Is a Power Grab", "mentions": 1}, {"url": "https://news.ycombinator.com/item?id=49630026", "title": "Qwen 3.8 follows GPT-5.5 Pro reasoning prefills", "mentions": 1}, {"url": "https://news.ycombinator.com/item?id=49611128", "title": "Benchmarking Qwen3.8 27B quantizations: 4-bit holds up, 1-bit collapses", "mentions": 1}, {"url": "https://news.ycombinator.com/item?id=49508982", "title": "Apple caught off guard by AI demand for Mac Mini and Mac Studio", "mentions": 1}, {"url": "https://news.ycombinator.com/item?id=49449507", "title": "GLM-5.3-Flash", "mentions": 1}], "sentiment_by_month": {"2025-11": {}, "2025-12": {}, "2026-01": {}, "2026-02": {"recommend": 2}, "2026-03": {"recommend": 1}, "2026-04": {"recommend": 3}, "2026-05": {}, "2026-06": {"recommend": 1, "warn": 2, "neutral": 2}, "2026-07": {"recommend": 1}, "2026-08": {"recommend": 1, "warn": 1, "neutral": 3}, "2026-09": {"neutral": 2, "recommend": 1}, "2026-10": {}}, "quotes": [{"id": 794592977, "item_id": 49662061, "url": "https://news.ycombinator.com/item?id=49662061", "by": "armchairhacker", "date": "2026-09-11", "stance": "recommend", "quote": "Unsloth (https://github.com/sponsors/unslothai)", "excerpt": "Even ignoring politics, I\u2019m sure there are more impactful ways to spend $18 million. Many OSS projects are underfunded, why not give some of that to e.g. Bevy (https://bevy.org/sponsorship-pledge/), Unsloth (https://github.com/sponsors/unslothai), or better, many small grants to many small projects?", "context": "", "use_case": "worthy OSS project to fund", "switched_from": "", "story": "Omarchy Is a Power Grab"}, {"id": 794129634, "item_id": 49633102, "url": "https://news.ycombinator.com/item?id=49633102", "by": "spijdar", "date": "2026-09-09", "stance": "neutral", "quote": "This is Unsloth's UD-Q4_K_S quantization", "excerpt": "This is Unsloth's UD-Q4_K_S quantization (edit -- on llama.cpp, via the Vulkan backend, on an RX 7900 XT, with Unsloth's recommended sampler config), for \"as replicable as LLMs can be\" disclosure, done through llama.cpp's web UI. In another thread, via Open WebUI (both with no custom prompt, so whatever the \"default\" system prompt is), a reasoning block opens as such: We need answer user's request: write complete sho", "context": "", "use_case": "quantized model weights", "switched_from": "", "story": "Qwen 3.8 follows GPT-5.5 Pro reasoning prefills"}, {"id": 793831312, "item_id": 49614457, "url": "https://news.ycombinator.com/item?id=49614457", "by": "mrbonner", "date": "2026-09-08", "stance": "neutral", "quote": "I use a 2-bit quant from Unsloth on my MBP M5 32GB of RAM.", "excerpt": "I use a 2-bit quant from Unsloth on my MBP M5 32GB of RAM. It run slower than molasses at 2 too/s kind of thing. Not sure it is usable at that rate for anything.", "context": "", "use_case": "running local quantized model", "switched_from": "", "story": "Benchmarking Qwen3.8 27B quantizations: 4-bit holds up, 1-bit collapses"}, {"id": 792240321, "item_id": 49515020, "url": "https://news.ycombinator.com/item?id=49515020", "by": "jckahn", "date": "2026-08-31", "stance": "neutral", "quote": "I use Unsloth's Q3 XXS with 128k context.", "excerpt": "I run Qwen 3.8 27B just fine on my Mac mini M4 24GB. I use Unsloth's Q3 XXS with 128k context. It successfully completes long horizon tasks with OpenCode.", "context": "", "use_case": "quantized local model", "switched_from": "", "story": "Apple caught off guard by AI demand for Mac Mini and Mac Studio"}, {"id": 791374225, "item_id": 49460889, "url": "https://news.ycombinator.com/item?id=49460889", "by": "Balinares", "date": "2026-08-27", "stance": "neutral", "quote": "I use the Unsloth UD_Q2_K_XL GGUF with default parameters", "excerpt": "I've found it tends toward long thinking loops even for simple tasks (and any quantization seems to increase their length), but those do exit eventually, unlike with Qwen 3.6. I use the Unsloth UD_Q2_K_XL GGUF with default parameters, along with that custom template linked elsewhere in the thread, and no K/V cache quantization.", "context": "", "use_case": "quantized local model", "switched_from": "", "story": "GLM-5.3-Flash"}, {"id": 791098034, "item_id": 49443627, "url": "https://news.ycombinator.com/item?id=49443627", "by": "cpburns2009", "date": "2026-08-26", "stance": "neutral", "quote": "I used unsloth's UD-Q{3-6}_K_XL quants across 11 evals.", "excerpt": "Not full precision. I've only benchmarked 27B across Q3-6 quants using lm-eval. I lack the hardware to bench 27B at BF16 but I might be able to do Q8_0. I haven't gotten around to doing 35B. I really should upload my collection of results to Github or somewhere. Here's a summary of what I have for 27B. I used unsloth's UD-Q{3-6}_K_XL quants across 11 evals. The values are pretty linear between Q3 and Q6. Qwen3.6-27B ", "context": "", "use_case": "quantized model weights", "switched_from": "", "story": "Qwen 3.8-Flash-Next releasing tomorrow (125B a6B)"}, {"id": 789912272, "item_id": 49369517, "url": "https://news.ycombinator.com/item?id=49369517", "by": "zenoprax", "date": "2026-08-19", "stance": "warn", "quote": "For some reason the unsloth models leave hardly any room for context.", "excerpt": "For some reason the unsloth models leave hardly any room for context. I've switched to the regular (non-unsloth) and get about 25 t/s and get about 80,000 more context tokens for the same quant.", "context": "", "use_case": "local quantized models", "switched_from": "", "story": "Unsloth Dynamic 3.0 GGUFs"}, {"id": 788508608, "item_id": 49281788, "url": "https://news.ycombinator.com/item?id=49281788", "by": "SwellJoe", "date": "2026-08-12", "stance": "recommend", "quote": "almost always, I pick the unsloth quantization if there is one", "excerpt": "Counterpoint: I've been using the Unsloth Gemma 4 quants extensively since very soon after release (on ROCm and Apple Silicon, I don't have any Nvidia hardware big enough), pretty much every quantization down to 4 bits (the QAT is the business, indistinguishable from the full-fat version, runs great on a slightly chonky desktop or laptop), and I haven't had any issues. The reason I use Gemma 4 so much often comes dow", "context": "Runs local models on ROCm and Apple Silicon", "use_case": "quantized local models", "switched_from": "", "story": "Qwen/Qwen3.8-2.4T-A95B"}, {"id": 780519682, "item_id": 48782480, "url": "https://news.ycombinator.com/item?id=48782480", "by": "SwellJoe", "date": "2026-07-03", "stance": "recommend", "quote": "if using dynamic quantizations like unsloth provides", "excerpt": "The Gemma 4 QAT 4-bit models are basically indistinguishable from the 8-bit or 16-bit versions. I tested, and many benchmarks agree. For models where there isn't a QAT version, 6-bit is where it starts to degrade, but only a little if using dynamic quantizations like unsloth provides. When I use Qwen, I use the 8-bit quantization. But, when I use Gemma there is literally no reason to use anything other than the 4-bit", "context": "", "use_case": "dynamic quantization", "switched_from": "", "story": "Kimi K2.7 Code is generally available in GitHub Copilot"}, {"id": 779619201, "item_id": 48726200, "url": "https://news.ycombinator.com/item?id=48726200", "by": "anon373839", "date": "2026-06-29", "stance": "neutral", "quote": "I\u2019m using Unsloth\u2019s Q8_K_XL quant", "excerpt": "I use Qwen 3.6 35B-A3B constantly, but I don\u2019t see the type of behavior you mentioned. I\u2019m using Unsloth\u2019s Q8_K_XL quant.", "context": "", "use_case": "quantized model weights", "switched_from": "", "story": "Qwen 3.6 27B is the sweet spot for local development"}, {"id": 779606529, "item_id": 48725408, "url": "https://news.ycombinator.com/item?id=48725408", "by": "cpburns2009", "date": "2026-06-29", "stance": "neutral", "quote": "I use unsloth's UD-Q5_K_XL at 256k context", "excerpt": "A 32gb card does run it nicely. I use unsloth's UD-Q5_K_XL at 256k context (k/v at q8_0), and get ~67 t/s on a 5090. I still need to look into MTP.", "context": "", "use_case": "quantized model weights", "switched_from": "", "story": "Qwen 3.6 27B is the sweet spot for local development"}, {"id": 778079521, "item_id": 48629970, "url": "https://news.ycombinator.com/item?id=48629970", "by": "xrd", "date": "2026-06-22", "stance": "warn", "quote": "it hacks a bunch of things to make it work and isn't optimal", "excerpt": "How are people running this locally? I just checked llama.cpp and it appears unsloth has a version but it hacks a bunch of things to make it work and isn't optimal. https://github.com/ggml-org/llama.cpp/issues/24730", "context": "", "use_case": "local model version", "switched_from": "", "story": "GLM 5.2 vs. Opus"}, {"id": 777952176, "item_id": 48622011, "url": "https://news.ycombinator.com/item?id=48622011", "by": "verdverm", "date": "2026-06-21", "stance": "warn", "quote": "They remained accusatory, made it personal, and then started deleting messages.", "excerpt": "Sure, I shared a project in their #research channel where I used their qwen36moe quant to refresh my PhD research. The channel had a topic that ended with something like \"and all things research...\" One of their people accused me of self-promotion, and I reiterated that I shared it in that channel because it was their quant doing something (I thought) interesting as a research model. They remained accusatory, made it", "context": "PhD researcher", "use_case": "quantized models for research", "switched_from": "", "story": "Two Qwen3 models on one DGX Spark: the residency math"}, {"id": 776919472, "item_id": 48557467, "url": "https://news.ycombinator.com/item?id=48557467", "by": "c0rruptbytes", "date": "2026-06-16", "stance": "recommend", "quote": "I recommend unsloth quants, i recommend 6bit for MoEs and 5bit for dense", "excerpt": "I don't know about good, I use a lot of local models and they're still pretty painful to run locally You have dense models (qwen 27b, gemma 31b) who are pretty smart, but pretty slow You have MoE models (gemma 26b, qwen 35b, north mini code 30b) who are pretty fast, but make a lot of mistakes You need a lot of memory to run these well, quantization makes tool calling weaker, so most run at 4 bit quants and are wonder", "context": "", "use_case": "quantized local models", "switched_from": "", "story": "Running local models is good now"}, {"id": 766822003, "item_id": 47926375, "url": "https://news.ycombinator.com/item?id=47926375", "by": "jonaustin", "date": "2026-04-27", "stance": "recommend", "quote": "make sure you're following Unsloth's recommendations for temperature/etc.", "excerpt": "Try https://github.com/jundot/omlx I've been using qwen3.5 (122b) with claude code for months, and it's definitely dumber than sonnet/opus, but it works through things reasonably well (i.e. writes half-decent code and tool calls usually work), and I pretty much never run into loops now. and make sure you're following Unsloth's recommendations for temperature/etc.", "context": "", "use_case": "local model settings", "switched_from": "", "story": "Running Local LLMs Offline on a Ten-Hour Flight"}, {"id": 766259153, "item_id": 47891197, "url": "https://news.ycombinator.com/item?id=47891197", "by": "mchusma", "date": "2026-04-24", "stance": "recommend", "quote": "Run through unsloth, and with some fine tuning", "excerpt": "At first, I was more excited about the Flash model, but I'm now more excited about the Pro model in many ways. I feel like the Pro model with an Run through unsloth, and with some fine tuning, is gonna be enough for many vertical SaaS applications. Where previously I was wary to under-provide the intelligence level, I'm now more excited about the idea of being able to give these pretty large intelligent models to my ", "context": "builds vertical SaaS applications", "use_case": "fine-tuning models", "switched_from": "", "story": "DeepSeek v4"}, {"id": 765996864, "item_id": 47874804, "url": "https://news.ycombinator.com/item?id=47874804", "by": "syntaxing", "date": "2026-04-23", "stance": "recommend", "quote": "I had similar issue until I used Unsloth\u2019s.", "excerpt": "Which quants are you using? I had similar issue until I used Unsloth\u2019s. I would recommend at least UD_6. Also, make sure your context length is above 65K. https://huggingface.co/unsloth/Qwen3.6-35B-A3B-GGUF", "context": "", "use_case": "quantized local Qwen models", "switched_from": "", "story": "Qwen3.6-27B: Flagship-Level Coding in a 27B Dense Model"}, {"id": 758733920, "item_id": 47420870, "url": "https://news.ycombinator.com/item?id=47420870", "by": "jacquesm", "date": "2026-03-17", "stance": "recommend", "quote": "Unsloth is the real thing. Highly recommended for those running their own AI engines", "excerpt": "Unsloth is the real thing. Highly recommended for those running their own AI engines and that want to get the most out of them.", "context": "", "use_case": "running local AI models", "switched_from": "", "story": "Unsloth Studio"}, {"id": 755129488, "item_id": 47195593, "url": "https://news.ycombinator.com/item?id=47195593", "by": "deepsquirrelnet", "date": "2026-02-28", "stance": "recommend", "quote": "I love the work unsloth is doing.", "excerpt": "I love the work unsloth is doing. I only wish gguf format had better vllm support. It\u2019s sometimes hard to find trustworthy quants that work well with vllm.", "context": "", "use_case": "quantized models", "switched_from": "", "story": "Unsloth Dynamic 2.0 GGUFs"}, {"id": 755067298, "item_id": 47191706, "url": "https://news.ycombinator.com/item?id=47191706", "by": "segmondy", "date": "2026-02-27", "stance": "recommend", "quote": "the unsloth quants are great", "excerpt": "This is a good time to promote running your own models. I have been running my own models locally and I would wager a local model will meet 85-95% of your needs if you really learn to use it. These models have gotten great. For anyone wanting to get into this, the smartest models to run recently that is consumer friendly was just released, checkout Qwen3.5 the 27B and 35B variants. They are small and I recommend runn", "context": "runs models locally", "use_case": "quantized model weights", "switched_from": "", "story": "How do I cancel my ChatGPT subscription?"}]}