{"slug": "apache-datafusion", "name": "Apache DataFusion", "category": "databases", "category_name": "Databases", "description": "", "recommenders": 3, "commenters": 13, "warners": 1, "score": 4.222, "rising": 0.35, "first": "2025-11-14", "latest": "2026-10-09", "why": "I thought of datafusion work and the rust implementation as a way to address the delta seen vs the Spark/JVM GraphFrames implementation", "why_is_quote": true, "has_verdict": false, "sentiment": {"recommend": 2, "neutral": 7, "compare": 3, "warn": 1}, "spark": [0, 0, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0], "listed": true, "loved": [], "caveat": "", "short_quote": "I thought of datafusion work and the rust implementation as a way to address the delta seen vs the Spark/JVM GraphFrames implementation", "kind": "", "icon": true, "domain": null, "github_repo": "github.com/apache/datafusion", "official_url": "https://datafusion.apache.org", "pricing": "", "aliases": ["Apache DataFusion", "DataFusion"], "verdict": null, "praise": [{"label": "", "summary": "", "quotes": [{"id": 791240194, "item_id": 49452512, "url": "https://news.ycombinator.com/item?id=49452512", "by": "willbeddow", "date": "2026-08-26", "stance": "recommend", "quote": "I remain excited about the broader ecosystem (especially datafusion).", "excerpt": "This is a real shame. I love DuckDB, and we've adopted/extended it and ducklake very heavily at Krea for our research. I know the projects will remain open, and ostensibly still contributed to in the same direction, but surely AWS thinks of this as another piece of a product suite to compete with Databricks. Nontheless, I think the open embedded query engine approach DuckDB is spearheading is larger than one project,", "context": "works at Krea", "use_case": "embedded query engine", "switched_from": "", "story": "AWS Acquires DuckLabs"}, {"id": 786062384, "item_id": 49128899, "url": "https://news.ycombinator.com/item?id=49128899", "by": "adsharma", "date": "2026-07-31", "stance": "recommend", "quote": "I thought of datafusion work and the rust implementation as a way to address the delta seen vs the Spark/JVM GraphFrames implementation", "excerpt": "There is a bit of healthy competition going on between graphframes and icebug. See: https://github.com/Ladybug-Memory/icebug-graphframes-comparison/ I thought of datafusion work and the rust implementation as a way to address the delta seen vs the Spark/JVM GraphFrames implementation (author is a major contributor). Looking forward to more such innovations, which will benefit the ecosystem as a whole. Why would anyon", "context": "", "use_case": "", "switched_from": "", "story": "Algorithms on billion-scale graph using 10GB RAM: I love DataFusion"}, {"id": 764217856, "item_id": 47763616, "url": "https://news.ycombinator.com/item?id=47763616", "by": "jeadie", "date": "2026-04-14", "stance": "recommend", "quote": "You might find https://github.com/apache/datafusion and https://github.com/datafusion-contrib/datafusion-federation of interest", "excerpt": "You might find https://github.com/apache/datafusion and https://github.com/datafusion-contrib/datafusion-federation of interest", "context": "", "use_case": "", "switched_from": "", "story": "Distributed DuckDB Instance"}]}], "caveats": [{"label": "", "summary": "", "quotes": [{"id": 751899746, "item_id": 46993734, "url": "https://news.ycombinator.com/item?id=46993734", "by": "thinkharderdev", "date": "2026-02-12", "stance": "warn", "quote": "I spent a very frustrating day tracking down this one in particular", "excerpt": "This will obviously depend on which implementation you use. Using the rust arrow-rs crate you at least get panics when you overflow max buffer sizes. But one of my enduring annoyances with arrow is that they use signed integer types for buffer offsets and the like. I understand why it has to be that way since it's intended to be cross-language and not all languages have unsigned integer types. But it does lead to lot", "context": "", "use_case": "", "switched_from": "", "story": "Apache Arrow is 10 years old"}]}], "who": [{"by": "willbeddow", "context": "works at Krea", "url": "https://news.ycombinator.com/item?id=49452512", "date": "2026-08-26"}], "tags": [], "use_cases": ["embedded query engine", "out-of-core graph storage", "sql query engine", "query planner reference", "big data processing"], "compared": [{"slug": "duckdb", "name": "DuckDB", "co": 3, "switched_from": 0, "switched_to": 0}, {"slug": "clickhouse", "name": "ClickHouse", "co": 2, "switched_from": 0, "switched_to": 0}, {"slug": "ducklake", "name": "DuckLake", "co": 2, "switched_from": 0, "switched_to": 0}, {"slug": "polars", "name": "Polars", "co": 2, "switched_from": 0, "switched_to": 0}, {"slug": "egui", "name": "egui", "co": 2, "switched_from": 0, "switched_to": 0}], "makers": [{"id": 802384321, "item_id": 50149020, "url": "https://news.ycombinator.com/item?id=50149020", "by": "gazpacho", "date": "2026-10-19", "stance": "neutral", "quote": "queried with DataFusion", "excerpt": "Author here. Very cool to see you commenting here, ClickHouse is such an amazing system! But yes, what's old is new. I didn't mean to claim we invented this technique. I actually cited ClickHouse's article (https://clickhouse.com/blog/a-new-powerful-json-data-type-for-clickhouse in the article) in the post : > Our shredded data layout predates both the Parquet Variant type and ClickHouse's JSON support but is very si", "context": "", "use_case": "Query engine for Parquet", "switched_from": "", "story": "Shredding JSON columns gave Logfire up to 1000x faster queries"}, {"id": 737845760, "item_id": 46115360, "url": "https://news.ycombinator.com/item?id=46115360", "by": "daguu", "date": "2025-12-01", "stance": "neutral", "quote": "integrate some of our data post-processing pipeline with our custom DataFusion UDF library", "excerpt": "Ursa Major | https://ursamajor.com | Entry-level Software Engineer | $95,000 - $130,000 Base | Full-time | Hybrid in-person in northern Colorado, US Ursa builds propulsion for aerospace and defense, so... rocket engines. My team works on software and process problems across the company, but we specialize in the data system used by development and test teams. We work on anything from data backhaul, post processing, an", "context": "Hiring manager at Ursa Major", "use_case": "Data processing pipeline", "switched_from": "", "story": "Ask HN: Who is hiring? (December 2025)"}], "threads": [{"url": "https://news.ycombinator.com/item?id=49124658", "title": "Algorithms on billion-scale graph using 10GB RAM: I love DataFusion", "mentions": 3}, {"url": "https://news.ycombinator.com/item?id=49448321", "title": "AWS Acquires DuckLabs", "mentions": 2}, {"url": "https://news.ycombinator.com/item?id=48085993", "title": "Ask HN: What Are You Working On? (May 2026)", "mentions": 2}, {"url": "https://news.ycombinator.com/item?id=50048490", "title": "Shredding JSON columns gave Logfire up to 1000x faster queries", "mentions": 1}, {"url": "https://news.ycombinator.com/item?id=49996149", "title": "DuckDB Ducklake", "mentions": 1}, {"url": "https://news.ycombinator.com/item?id=49686380", "title": "Ask HN: What are you working on? (September 2026)", "mentions": 1}], "sentiment_by_month": {"2025-11": {"compare": 1}, "2025-12": {}, "2026-01": {}, "2026-02": {"warn": 1}, "2026-03": {"neutral": 1, "compare": 1}, "2026-04": {"recommend": 1}, "2026-05": {"neutral": 2}, "2026-06": {}, "2026-07": {"compare": 1}, "2026-08": {"neutral": 2, "recommend": 1}, "2026-09": {"neutral": 1}, "2026-10": {"neutral": 1}}, "quotes": [{"id": 800247009, "item_id": 50015438, "url": "https://news.ycombinator.com/item?id=50015438", "by": "eddietejeda", "date": "2026-10-09", "stance": "neutral", "quote": "bringing-ducklake-to-datafusion", "excerpt": "Extremely high concurrency at sub-second response times. Here is our write up on the Ducklake blog: https://ducklake.select/2026/07/29/bringing-ducklake-to-datafusion/", "context": "", "use_case": "", "switched_from": "", "story": "DuckDB Ducklake"}, {"id": 795190819, "item_id": 49699426, "url": "https://news.ycombinator.com/item?id=49699426", "by": "lvduan", "date": "2026-09-14", "stance": "neutral", "quote": "the rest uses Apache Data Fusion Parser", "excerpt": "I built this sql formatter <https://sql.leuduan.work> for my own use - it works by parsing the SQL query to AST using native engines where possible (for BigQuery it use GoogleSQL library, for MSSQL it uses Microsoft ScriptDOM, the rest uses Apache Data Fusion Parser) and from the AST it re-generates the SQL query in a consistent formatted style. You can throw in all kind of badly formatted SQL queries and it will wor", "context": "", "use_case": "", "switched_from": "", "story": "Ask HN: What are you working on? (September 2026)"}, {"id": 791240194, "item_id": 49452512, "url": "https://news.ycombinator.com/item?id=49452512", "by": "willbeddow", "date": "2026-08-26", "stance": "recommend", "quote": "I remain excited about the broader ecosystem (especially datafusion).", "excerpt": "This is a real shame. I love DuckDB, and we've adopted/extended it and ducklake very heavily at Krea for our research. I know the projects will remain open, and ostensibly still contributed to in the same direction, but surely AWS thinks of this as another piece of a product suite to compete with Databricks. Nontheless, I think the open embedded query engine approach DuckDB is spearheading is larger than one project,", "context": "works at Krea", "use_case": "embedded query engine", "switched_from": "", "story": "AWS Acquires DuckLabs"}, {"id": 791190688, "item_id": 49449418, "url": "https://news.ycombinator.com/item?id=49449418", "by": "tobilg", "date": "2026-08-26", "stance": "neutral", "quote": "You can even use DuckLake with Datafusion", "excerpt": "You can even use DuckLake with Datafusion: https://github.com/tobilg/datafusion-ducklake-provider", "context": "", "use_case": "", "switched_from": "", "story": "AWS Acquires DuckLabs"}, {"id": 786399728, "item_id": 49149983, "url": "https://news.ycombinator.com/item?id=49149983", "by": "stuhood", "date": "2026-08-02", "stance": "neutral", "quote": "Co-partitioning of range-partitioned data is now supported on `main` too", "excerpt": "Co-partitioning of range-partitioned data is now supported on `main` too: see https://github.com/apache/datafusion/issues/23183 and the epic https://github.com/apache/datafusion/issues/22395", "context": "", "use_case": "", "switched_from": "", "story": "Algorithms on billion-scale graph using 10GB RAM: I love DataFusion"}, {"id": 786062384, "item_id": 49128899, "url": "https://news.ycombinator.com/item?id=49128899", "by": "adsharma", "date": "2026-07-31", "stance": "recommend", "quote": "I thought of datafusion work and the rust implementation as a way to address the delta seen vs the Spark/JVM GraphFrames implementation", "excerpt": "There is a bit of healthy competition going on between graphframes and icebug. See: https://github.com/Ladybug-Memory/icebug-graphframes-comparison/ I thought of datafusion work and the rust implementation as a way to address the delta seen vs the Spark/JVM GraphFrames implementation (author is a major contributor). Looking forward to more such innovations, which will benefit the ecosystem as a whole. Why would anyon", "context": "", "use_case": "", "switched_from": "", "story": "Algorithms on billion-scale graph using 10GB RAM: I love DataFusion"}, {"id": 786039808, "item_id": 49127488, "url": "https://news.ycombinator.com/item?id=49127488", "by": "adsharma", "date": "2026-07-31", "stance": "compare", "quote": "datafusion allows you to do fine grained storage integration", "excerpt": "https://github.com/LadybugDB/ladybug-icebug-notebooks/blob/main/ladybug_icebug_comm_detection.ipynb Trade-off: datafusion allows you to do fine grained storage integration (spill to disk as a part of the algorithm). The icebug/ladybug way is coarse grained. But it allows you to run cypher instead of writing datafusion operators.", "context": "", "use_case": "out-of-core graph storage", "switched_from": "", "story": "Algorithms on billion-scale graph using 10GB RAM: I love DataFusion"}, {"id": 769961297, "item_id": 48122581, "url": "https://news.ycombinator.com/item?id=48122581", "by": "notfirstpost", "date": "2026-05-13", "stance": "neutral", "quote": "Apache DataFusion for SQL engine", "excerpt": "Hi, I'm JC and I built DAGraph over the past few months. It is a visual, graph-based, local-first data web-app, to do computations from simple arithmetic to advanced OLAP SQL. There's no account or sign up needed to use it. Your data stays in the browser, no upload. Some technical aspects that might be of interest here: - written in Rust targeting WASM with Trunk; - Apache DataFusion for SQL engine + Arrow + Parquet;", "context": "", "use_case": "", "switched_from": "", "story": "Show HN: DAGraph \u2013 local-first reactive graph from arithmetic to OLAP SQL"}, {"id": 769492225, "item_id": 48093264, "url": "https://news.ycombinator.com/item?id=48093264", "by": "notfirstpost", "date": "2026-05-11", "stance": "neutral", "quote": "The SQL engine is Apache DataFusion[1]", "excerpt": "I am working on DAGraph (https://dagraph.com) DAGraph is a local-first reactive DAG for analytical SQL (OLAP), running entirely in the browser (there is also a native version in the works). Some tech details: written in Rust, targeting WASM (and native). The SQL engine is Apache DataFusion[1]. The UI uses Egui[2]. Workspace data is persisted in browser using OPFS[3] via OpenDAL[4]. The graph is functional and handles", "context": "", "use_case": "SQL query engine", "switched_from": "", "story": "Ask HN: What Are You Working On? (May 2026)"}, {"id": 769389280, "item_id": 48086830, "url": "https://news.ycombinator.com/item?id=48086830", "by": "jesol", "date": "2026-05-10", "stance": "neutral", "quote": "spelunk in the Data Fusion codebase", "excerpt": "I'm building a CRDT replicated EAV graph store with reactive datalog optimized with a cascades query planner, and uses random walks during planning of the data on disk (assume NVMEs) for cardinality estimation. The idea is to provide a structured knowledge base that can quickly run queries on relatively small data (10m-100m facts) to an LLM, and teach the model to always do research within the knowledge base using po", "context": "", "use_case": "Query planner reference", "switched_from": "", "story": "Ask HN: What Are You Working On? (May 2026)"}, {"id": 764217856, "item_id": 47763616, "url": "https://news.ycombinator.com/item?id=47763616", "by": "jeadie", "date": "2026-04-14", "stance": "recommend", "quote": "You might find https://github.com/apache/datafusion and https://github.com/datafusion-contrib/datafusion-federation of interest", "excerpt": "You might find https://github.com/apache/datafusion and https://github.com/datafusion-contrib/datafusion-federation of interest", "context": "", "use_case": "", "switched_from": "", "story": "Distributed DuckDB Instance"}, {"id": 756270049, "item_id": 47266878, "url": "https://news.ycombinator.com/item?id=47266878", "by": "nlz", "date": "2026-03-05", "stance": "compare", "quote": "DataFusion as the more complete option", "excerpt": "Yes! This is a great idea. Thanks for mentioning it! GlueSQL could be another cool target too (https://github.com/gluesql/gluesql). I think there's a fun exploration in taking a storage engine and seeing if it's compatible with different SQL layers \u2014 GlueSQL as a simpler starting point, DataFusion as the more complete option (https://github.com/apache/datafusion). Plugging into a real SQL engine seems like a great st", "context": "", "use_case": "", "switched_from": "", "story": "Show HN: I built an LSM storage engine from scratch in Rust"}, {"id": 756062516, "item_id": 47253907, "url": "https://news.ycombinator.com/item?id=47253907", "by": "vishinvents", "date": "2026-03-04", "stance": "neutral", "quote": "Stack is Iggy, Polars, Datafusion, RisingWave, R2, CF Workers, Civo", "excerpt": "Hi all, Been a lurker here for long, figured I'd share something that could be useful for some. I've vibecoded a GA alternative over a weekend, but really it's more like 'define any events' and query them later type processor. Really disliked the complexity of GA's setup, especially around ad attribution. I have extensive experience in AdTech, so I'm open to hear what ya'll think too. Stack is Iggy, Polars, Datafusio", "context": "", "use_case": "", "switched_from": "", "story": "Show HN: Gobble \u2013 Yet Another OSS Alternative to Google Analytics/PostHog, etc."}, {"id": 751899746, "item_id": 46993734, "url": "https://news.ycombinator.com/item?id=46993734", "by": "thinkharderdev", "date": "2026-02-12", "stance": "warn", "quote": "I spent a very frustrating day tracking down this one in particular", "excerpt": "This will obviously depend on which implementation you use. Using the rust arrow-rs crate you at least get panics when you overflow max buffer sizes. But one of my enduring annoyances with arrow is that they use signed integer types for buffer offsets and the like. I understand why it has to be that way since it's intended to be cross-language and not all languages have unsigned integer types. But it does lead to lot", "context": "", "use_case": "", "switched_from": "", "story": "Apache Arrow is 10 years old"}, {"id": 734822726, "item_id": 45926420, "url": "https://news.ycombinator.com/item?id=45926420", "by": "faizshah", "date": "2025-11-14", "stance": "compare", "quote": "I also tried Datafusion and polars but clickhouse ended up being faster.", "excerpt": "I had to do something like this for a few TB of json recently. The unique thing about this workload was it was a ton of small 10-20mb files. I found that clickhouse was the fastest, but duckdb was the simplest to work with it usually just works. DuckDB was close enough to the max performance from clickhouse. I tried flink & pyspark but they were way slower (like 3-5x) than clickhouse and the code was kind of annoying", "context": "", "use_case": "big data processing", "switched_from": "", "story": "650GB of Data (Delta Lake on S3). Polars vs. DuckDB vs. Daft vs. Spark"}]}