{"slug": "openai-tokenizer", "name": "OpenAI Tokenizer", "category": "other", "category_name": "Other software", "description": "", "recommenders": 3, "commenters": 4, "warners": 0, "score": 2.971, "rising": -0.98, "first": "2026-02-26", "latest": "2026-07-30", "why": "Great for getting particular about how to save tokens in prompt and tool definitions", "why_is_quote": true, "has_verdict": false, "sentiment": {"recommend": 3, "neutral": 1, "compare": 0, "warn": 0}, "spark": [0, 0, 0, 1, 0, 0, 1, 0, 1, 0, 0, 0], "listed": true, "loved": [], "caveat": "", "short_quote": "Great for getting particular about how to save tokens in prompt and tool definitions", "kind": "", "icon": true, "domain": "openai.com", "github_repo": null, "official_url": "https://platform.openai.com/tokenizer", "pricing": "", "aliases": ["OpenAI Tokenizer"], "verdict": null, "praise": [{"label": "", "summary": "", "quotes": [{"id": 785905920, "item_id": 49119120, "url": "https://news.ycombinator.com/item?id=49119120", "by": "Townley", "date": "2026-07-30", "stance": "recommend", "quote": "Great for getting particular about how to save tokens in prompt and tool definitions", "excerpt": "Using Chinese could lead to a small amount of gains, but it's nowhere near as drastic as you'd think Putting aside the question of whether English's larger training corpus would be a dimension of quality: content is \"tokenized\" before the model sees it. The \"~4 English chars/token\" rule isn't a strategy to optimize around because in practice, many English words become one token, the same way Chinese words do. \"I love", "context": "", "use_case": "inspecting token counts", "switched_from": "", "story": "Does Speaking to Agents Like Cavemen Save 65% of Tokens? We Test"}, {"id": 754611952, "item_id": 47163247, "url": "https://news.ycombinator.com/item?id=47163247", "by": "eigenspace", "date": "2026-02-26", "stance": "recommend", "quote": "You can play around with how text gets tokenized here", "excerpt": "They're text generators, but you can think of them as basically operating with a different alphabet than us. When they are given text input, it's not in our alphabet, and when they produce text output it's also not in our alphabet. So when you ask them what letters are in a given word, they're literally just guessing when they respond. Rather, they use tokens that are usually combinations of 2-8 characters. You can p", "context": "", "use_case": "explore text tokenization", "switched_from": "", "story": "How Will OpenAI Compete?"}]}], "caveats": [], "who": [], "tags": [], "use_cases": ["inspecting token counts", "token counting", "explore text tokenization"], "compared": [], "makers": [], "threads": [{"url": "https://news.ycombinator.com/item?id=49080605", "title": "Does Speaking to Agents Like Cavemen Save 65% of Tokens? We Test", "mentions": 1}, {"url": "https://news.ycombinator.com/item?id=48270770", "title": "Norway's 2 petabytes of Huawei flash storage and LLM training", "mentions": 1}, {"url": "https://news.ycombinator.com/item?id=48174920", "title": "How fast is N tokens per second really?", "mentions": 1}, {"url": "https://news.ycombinator.com/item?id=47158975", "title": "How Will OpenAI Compete?", "mentions": 1}], "sentiment_by_month": {"2025-11": {}, "2025-12": {}, "2026-01": {}, "2026-02": {"recommend": 1}, "2026-03": {}, "2026-04": {}, "2026-05": {"recommend": 1, "neutral": 1}, "2026-06": {}, "2026-07": {"recommend": 1}, "2026-08": {}, "2026-09": {}, "2026-10": {}}, "quotes": [{"id": 785905920, "item_id": 49119120, "url": "https://news.ycombinator.com/item?id=49119120", "by": "Townley", "date": "2026-07-30", "stance": "recommend", "quote": "Great for getting particular about how to save tokens in prompt and tool definitions", "excerpt": "Using Chinese could lead to a small amount of gains, but it's nowhere near as drastic as you'd think Putting aside the question of whether English's larger training corpus would be a dimension of quality: content is \"tokenized\" before the model sees it. The \"~4 English chars/token\" rule isn't a strategy to optimize around because in practice, many English words become one token, the same way Chinese words do. \"I love", "context": "", "use_case": "inspecting token counts", "switched_from": "", "story": "Does Speaking to Agents Like Cavemen Save 65% of Tokens? We Test"}, {"id": 772410864, "item_id": 48275679, "url": "https://news.ycombinator.com/item?id=48275679", "by": "YetAnotherNick", "date": "2026-05-25", "stance": "neutral", "quote": "I tested it on few translations on wikipedia articles using", "excerpt": "Did you even try to verify your claims. I tested it on few translations on wikipedia articles using [1] and it takes 15-20% more tokens for Norwegian. English performs the best because there is more data in English and high quality sources are either only in English or there is a good translation in English. [1]: https://platform.openai.com/tokenizer", "context": "", "use_case": "", "switched_from": "", "story": "Norway's 2 petabytes of Huawei flash storage and LLM training"}, {"id": 771465728, "item_id": 48216608, "url": "https://news.ycombinator.com/item?id=48216608", "by": "davely", "date": "2026-05-20", "stance": "recommend", "quote": "", "excerpt": "I built something similar awhile back [1] and used OpenAI\u2019s tokenizer playground [2] to recalculate tokens on a giant block of lorem ipsum text. I feel like this gives a much more accurate representation. [1] https://dave.ly/tools/tokenflow/ [2] https://platform.openai.com/tokenizer", "context": "", "use_case": "token counting", "switched_from": "", "story": "How fast is N tokens per second really?"}, {"id": 754611952, "item_id": 47163247, "url": "https://news.ycombinator.com/item?id=47163247", "by": "eigenspace", "date": "2026-02-26", "stance": "recommend", "quote": "You can play around with how text gets tokenized here", "excerpt": "They're text generators, but you can think of them as basically operating with a different alphabet than us. When they are given text input, it's not in our alphabet, and when they produce text output it's also not in our alphabet. So when you ask them what letters are in a given word, they're literally just guessing when they respond. Rather, they use tokens that are usually combinations of 2-8 characters. You can p", "context": "", "use_case": "explore text tokenization", "switched_from": "", "story": "How Will OpenAI Compete?"}]}