GenDB is a Generative Query Engine that uses LLM agents to generate instance-optimized
query execution code, tailored to your specific data, workloads, and hardware.
Five specialized LLM agents collaborate through a structured pipeline to generate
optimized storage, indexes, and standalone native executables — all tailored to the specific data, workload, and hardware.
Profiles hardware, samples data, extracts workload characteristics
Designs layouts with encoding, compression, indexes, and zone maps
Generates resource-aware execution plans adapted to data and hardware
Implements plans as optimized native code with SIMD and parallelism
Iteratively refines code using runtime profiling feedback
Today, every new use case demands either a painful extension or an entirely new system:
PostgreSQL → PostGIS, TimescaleDB, pgvector, Citus, AGE …
Each extension fights the host system’s architectural constraints.
DuckDB, Umbra, ClickHouse, Milvus, Pinecone, InfluxDB, Neo4j …
Each requires years of engineering and huge monetary costs.
Use LLMs to generate per-query execution code. No extension wrestling, no multi-year engineering. New techniques become reachable through prompt updates.
Instance-optimized code exploits exact data distributions, join selectivities, group cardinalities, and hardware characteristics. No general-purpose engine can match this.
Integrating new techniques requires prompting, not re-engineering. Semantic queries, GPU-native code — all reachable through prompt updates.
80% of queries repeat in 50% of clusters. Generation cost is amortized over many executions, making it cost-effective for recurring analytical workloads.
Total query execution time across all queries. GenDB variants use different LLM backbone models.
All systems run on identical hardware with full parallelism enabled.
Different LLM backbone models offer different trade-offs between generated code quality, generation time, and cost. Ranked by average query execution time.
We select the best-performing C++ binary for each TPC-H query from a GenDB run, then give
Claude Code (Opus 4.6) 5 iterations to analyze, profile, and improve —
first for optimized C++, then for a full Rust rewrite.
GenDB-generated code with standard compilation.
Aggressive flags, madvise tuning, parallelized joins, thread optimization.
Full rewrite with rayon, memmap2, unsafe bounds-check elimination.
Key findings: Optimized C++ achieves a 1.30x overall speedup, with Q18 showing the largest gain (2.44x) from parallelized join building.
Rust wins on Q6 (zone-map scan with get_unchecked) but carries ~30ms per-query overhead from mmap page table setup, penalizing short queries.
The Rust main_scan compute times are competitive with C++, suggesting the overhead is structural rather than algorithmic.
We plan to introduce a dedicated Code Refiner agent to the pipeline, responsible for low-level, implementation-level optimizations — to automatically achieve these gains as part of the standard GenDB workflow.
GenDB is under active development. Every step follows three principles:
Multi-agent pipeline for analytical queries. Evaluated on TPC-H and SEC-EDGAR, outperforming DuckDB, Umbra, ClickHouse, MonetDB, and PostgreSQL.
Agents learn from past runs, accumulate optimization experience, and improve generation quality over time — without retraining the underlying LLMs.
Generate CUDA and GPU-accelerated code targeting libcudf for cost-efficient GPU analytics, not just CPU.
Generate code for multimodal data — images, audio, text — with AI-powered operators, moving beyond SQL’s relational model.
Reusable operators across queries, query template generation, hybrid execution with traditional DBMS, and further cost reduction as LLMs become faster and cheaper.
Link preview
GenDB: The Next Generation of Query Processing -- Synthesized, Not Engineered
Traditional query processing relies on engines that are carefully optimized and engineered by many experts. However, new techniques and user requirements evolve rapidly, and existing systems often cannot keep pace. At the same time, these systems are difficult to extend due to their internal complexity, and developing new systems requires substantial engineering effort and cost. In this paper, we argue that recent advances in Large Language Models (LLMs) are starting to shape the next generation of query processing systems. We propose using LLMs to synthesize execution code for each incoming query, instead of continuously building, extending, and maintaining complex query processing engines. As a proof of concept, we present GenDB, an LLM-powered agentic system that generates instance-optimized and customized query execution code tailored to specific data, workloads, and hardware resources. We implemented an early prototype of GenDB that uses Claude Code Agent as the underlying component in the multi-agent system, and we evaluate it on OLAP workloads. We use queries from the well-known TPC-H benchmark and also construct a new benchmark designed to reduce potential data leakage from LLM training data. We compare GenDB with state-of-the-art query engines, including DuckDB, Umbra, MonetDB, ClickHouse, and PostgreSQL. GenDB achieves significantly better performance than these systems. Finally, we discuss the current limitations of GenDB and outline future extensions and related research challenges. solidlao.github.io · arxiv.org
Comments