{"slug": "promptfoo", "name": "promptfoo", "category": "ai", "category_name": "AI models and assistants", "description": "", "recommenders": 4, "commenters": 6, "warners": 0, "score": 4.04, "rising": -1.31, "first": "2026-01-31", "latest": "2026-08-21", "why": "I used Promptpoo at the beginning of the year - it was exactly, what I needed", "why_is_quote": true, "has_verdict": false, "sentiment": {"recommend": 4, "neutral": 1, "compare": 1, "warn": 0}, "spark": [0, 0, 1, 0, 1, 0, 0, 1, 1, 0, 0, 0], "listed": true, "loved": [], "caveat": "", "short_quote": "I used Promptpoo at the beginning of the year - it was exactly, what I needed", "kind": "", "icon": true, "domain": null, "github_repo": "github.com/promptfoo/promptfoo", "official_url": "https://www.promptfoo.dev", "pricing": "", "aliases": ["promptfoo", "Promptfoo"], "verdict": null, "praise": [{"label": "", "summary": "", "quotes": [{"id": 785564192, "item_id": 49097762, "url": "https://news.ycombinator.com/item?id=49097762", "by": "_the_inflator", "date": "2026-07-29", "stance": "recommend", "quote": "I used Promptpoo at the beginning of the year - it was exactly, what I needed", "excerpt": "Off topic: Just some thx and kudos to you guys. I used Promptpoo at the beginning of the year - it was exactly, what I needed, very much still a niche thing hardly anyone was using. I totally missed the acquisition - but well deserved. I am currently re-evaluating PF again for my upcoming project, and happy to see that it is more than simply thriving.", "context": "", "use_case": "LLM prompt evaluation", "switched_from": "", "story": "OpenAI just open-sourced Codex Security"}, {"id": 776890144, "item_id": 48555634, "url": "https://news.ycombinator.com/item?id=48555634", "by": "rhdunn", "date": "2026-06-16", "stance": "recommend", "quote": "I use promptfoo for evaluation.", "excerpt": "I use promptfoo for evaluation. I'm experimenting with tests for my workflow/use cases. I have a custom assert for loop/repeat detection that works well: def count_repeats(text: str, length: int) -> int: n = len(text) pattern = text[n - length : n] count = 1 # Include the end of the string as matching the substring. text = text[: -length] while text.endswith(pattern): text = text[: -length] count = count + 1 return c", "context": "", "use_case": "LLM evaluation", "switched_from": "", "story": "Ask HN: Has anyone replaced Claude/GPT with a local model for daily coding?"}, {"id": 757177152, "item_id": 47323572, "url": "https://news.ycombinator.com/item?id=47323572", "by": "benban", "date": "2026-03-10", "stance": "recommend", "quote": "my experience with Promptfoo is great at testing model outputs", "excerpt": "This makes sense for OpenAI, my experience with Promptfoo is great at testing model outputs. But I keep wondering who's looking at the other side: the actual agent code, and what happens now for other models such as Gemini/Claude etc that are using Promptfoo being locked-in with OpenAI and OS. Like, an eval will tell you the model gave a bad answer. It won't tell you that your agent passes that answer straight into a", "context": "", "use_case": "LLM output evaluation", "switched_from": "", "story": "OpenAI Acquires Promptfoo"}, {"id": 749486752, "item_id": 46842922, "url": "https://news.ycombinator.com/item?id=46842922", "by": "alexhans", "date": "2026-01-31", "stance": "recommend", "quote": "promptfoo (Technically no programming required for an eval creator, just yaml)", "excerpt": "> Curious what peeple are doing in the real world It's hard to answer this as the right thing to do might depend a lot on the environment. My pitch would be that automation is not a new problem and the fundamentals still apply: - We can\u2019t compare what we can\u2019t measure - I need to ask myself, can I trust this to run on its own? - I need to be able to describe what I want - I need to understand my risk profile Once you", "context": "", "use_case": "", "switched_from": "", "story": "Evaluations for Testing Agentic AI"}]}], "caveats": [], "who": [], "tags": [], "use_cases": ["llm evals", "llm prompt evaluation", "llm evaluation", "llm output evaluation"], "compared": [], "makers": [{"id": 785442897, "item_id": 49090181, "url": "https://news.ycombinator.com/item?id=49090181", "by": "dangelosaurus", "date": "2026-07-28", "stance": "neutral", "quote": "co-founder of Promptfoo", "excerpt": "Hey HN, Michael here, co-founder of Promptfoo and one of the people working on the Codex Security CLI at OpenAI. Thanks for checking this out and for flagging the auth issues. We just open-sourced it, and there's still plenty for us to improve. Expect the product to evolve quickly. If you try it, I'd really appreciate hearing what works well and what you think we should improve. Happy to answer questions here. CLI do", "context": "", "use_case": "", "switched_from": "", "story": "OpenAI just open-sourced Codex Security"}, {"id": 773761105, "item_id": 48360069, "url": "https://news.ycombinator.com/item?id=48360069", "by": "b00gn1sh", "date": "2026-06-01", "stance": "recommend", "quote": "Promptfoo eval gates blocking bad output before it persists", "excerpt": "Location: Chattanooga, TN, USA Remote: Yes Willing to relocate: No Technologies: Swift / SwiftUI (native macOS), production LLM pipelines (Whisper, Llama 3.3, GLM-5.1, OpenAI / Anthropic APIs), Promptfoo eval suites, Postgres + queue-based job infrastructure, self-hosted GPU (RTX 3060), Figma, Next.js / Vercel. R\u00e9sum\u00e9/CV: https://doubleabattery.com/work-with-me Email: hello@doubleabattery.com 25-year design-led produ", "context": "", "use_case": "LLM evaluation", "switched_from": "", "story": "Ask HN: Who wants to be hired? (June 2026)"}, {"id": 740270928, "item_id": 46266933, "url": "https://news.ycombinator.com/item?id=46266933", "by": "dangelosaurus", "date": "2025-12-14", "stance": "recommend", "quote": "Working on promptfoo", "excerpt": "Working on promptfoo, an open-source (MIT) CLI and framework for eval-ing and red-teaming LLM apps. Think of it like pytest but for prompts - you define test cases, run evals against any model (OpenAI, Anthropic, local models, whatever), and catch regressions before they hit prod. Currently building out support for multi-agent evals, better tracing, voice, and static code analysis for AI security use cases. So many f", "context": "", "use_case": "LLM eval and red-teaming CLI", "switched_from": "", "story": "Ask HN: What Are You Working On? (December 2025)"}], "threads": [{"url": "https://news.ycombinator.com/item?id=49089755", "title": "OpenAI just open-sourced Codex Security", "mentions": 2}, {"url": "https://news.ycombinator.com/item?id=49393051", "title": "A week of using Codex more than Claude", "mentions": 1}, {"url": "https://news.ycombinator.com/item?id=49332751", "title": "Launch HN: Speko (YC S26) \u2013 OpenRouter for Voice AI", "mentions": 1}, {"url": "https://news.ycombinator.com/item?id=48542100", "title": "Ask HN: Has anyone replaced Claude/GPT with a local model for daily coding?", "mentions": 1}, {"url": "https://news.ycombinator.com/item?id=48357724", "title": "Ask HN: Who wants to be hired? (June 2026)", "mentions": 1}, {"url": "https://news.ycombinator.com/item?id=47323527", "title": "OpenAI Acquires Promptfoo", "mentions": 1}], "sentiment_by_month": {"2025-11": {}, "2025-12": {}, "2026-01": {"recommend": 1}, "2026-02": {}, "2026-03": {"recommend": 1}, "2026-04": {}, "2026-05": {}, "2026-06": {"recommend": 1}, "2026-07": {"recommend": 1}, "2026-08": {"compare": 1, "neutral": 1}, "2026-09": {}, "2026-10": {}}, "quotes": [{"id": 790305906, "item_id": 49394119, "url": "https://news.ycombinator.com/item?id=49394119", "by": "azuanrb", "date": "2026-08-21", "stance": "neutral", "quote": "I also have some custom evals using promptfoo to make sure I\u2019m not introducing regressions when switching models.", "excerpt": "I\u2019ve been experimenting with this for a while, and right now I\u2019m using Luna xhigh as my default. Previously I was using Sol medium. Sol medium is a great balance between speed and being thorough, but it\u2019s quite expensive. Luna xhigh seems to compensate for slightly lower intelligence by thinking and reasoning for longer, so tasks can take more time to complete. But it\u2019s crazy cheap. I also have some custom evals usin", "context": "", "use_case": "LLM evals", "switched_from": "", "story": "A week of using Codex more than Claude"}, {"id": 789343905, "item_id": 49333994, "url": "https://news.ycombinator.com/item?id=49333994", "by": "abdik", "date": "2026-08-17", "stance": "compare", "quote": "\"promptfoo of voice models\": that is closer to how it started.", "excerpt": "Fair pushback. On end to end: we measure those too, same methodology: https://benchmarks.speko.ai/s2s. If the single models win, we route to them the same way, so we do not care which architecture (s2s or cascaded) wins. For now, what we see in production so far is that most teams still want to control each piece: swap the STT for medical vocabulary, keep the LLM, keep the voice. On \"promptfoo of voice models\": that ", "context": "", "use_case": "", "switched_from": "", "story": "Launch HN: Speko (YC S26) \u2013 OpenRouter for Voice AI"}, {"id": 785564192, "item_id": 49097762, "url": "https://news.ycombinator.com/item?id=49097762", "by": "_the_inflator", "date": "2026-07-29", "stance": "recommend", "quote": "I used Promptpoo at the beginning of the year - it was exactly, what I needed", "excerpt": "Off topic: Just some thx and kudos to you guys. I used Promptpoo at the beginning of the year - it was exactly, what I needed, very much still a niche thing hardly anyone was using. I totally missed the acquisition - but well deserved. I am currently re-evaluating PF again for my upcoming project, and happy to see that it is more than simply thriving.", "context": "", "use_case": "LLM prompt evaluation", "switched_from": "", "story": "OpenAI just open-sourced Codex Security"}, {"id": 776890144, "item_id": 48555634, "url": "https://news.ycombinator.com/item?id=48555634", "by": "rhdunn", "date": "2026-06-16", "stance": "recommend", "quote": "I use promptfoo for evaluation.", "excerpt": "I use promptfoo for evaluation. I'm experimenting with tests for my workflow/use cases. I have a custom assert for loop/repeat detection that works well: def count_repeats(text: str, length: int) -> int: n = len(text) pattern = text[n - length : n] count = 1 # Include the end of the string as matching the substring. text = text[: -length] while text.endswith(pattern): text = text[: -length] count = count + 1 return c", "context": "", "use_case": "LLM evaluation", "switched_from": "", "story": "Ask HN: Has anyone replaced Claude/GPT with a local model for daily coding?"}, {"id": 757177152, "item_id": 47323572, "url": "https://news.ycombinator.com/item?id=47323572", "by": "benban", "date": "2026-03-10", "stance": "recommend", "quote": "my experience with Promptfoo is great at testing model outputs", "excerpt": "This makes sense for OpenAI, my experience with Promptfoo is great at testing model outputs. But I keep wondering who's looking at the other side: the actual agent code, and what happens now for other models such as Gemini/Claude etc that are using Promptfoo being locked-in with OpenAI and OS. Like, an eval will tell you the model gave a bad answer. It won't tell you that your agent passes that answer straight into a", "context": "", "use_case": "LLM output evaluation", "switched_from": "", "story": "OpenAI Acquires Promptfoo"}, {"id": 749486752, "item_id": 46842922, "url": "https://news.ycombinator.com/item?id=46842922", "by": "alexhans", "date": "2026-01-31", "stance": "recommend", "quote": "promptfoo (Technically no programming required for an eval creator, just yaml)", "excerpt": "> Curious what peeple are doing in the real world It's hard to answer this as the right thing to do might depend a lot on the environment. My pitch would be that automation is not a new problem and the fundamentals still apply: - We can\u2019t compare what we can\u2019t measure - I need to ask myself, can I trust this to run on its own? - I need to be able to describe what I want - I need to understand my risk profile Once you", "context": "", "use_case": "", "switched_from": "", "story": "Evaluations for Testing Agentic AI"}]}