Zilliz Adds Cost-Aware Benchmarking to VDBBench, the Open-Source Vector Database Benchmark

Article content

Article content

VDBBench already tests vector databases under real production workloads; The new release lets any team measure what a vector database actually costs to run in production — across data freshness, filtering, multitenancy, and cold starts — not just peak queries per second.

Article content

Article content

REDWOOD SHORES, Calif. — Zilliz, a leading AI data infrastructure company and the creator of Milvus, recently announced a major update to VectorDBBench (VDBBench), an open-source, vendor-neutral benchmark for vector databases, adding cost as a first-class dimension alongside performance under production conditions — not just how fast a system performs in an idealized test.

Article content

Article content

Most benchmarks optimize for a single headline number: peak queries per second (QPS) on static, fully indexed data. But teams rarely choose a vector database on speed alone. They need to know what it costs to hit a target QPS, when newly written data actually becomes searchable, how filters and payload size change the query surface, how a system holds up across thousands of tenants, and how it responds on the first query after sitting idle. VDBBench now measures those dimensions directly.

Article content

Article content

“Most benchmarks answer one question — how fast can it go on ideal data,” said James Luan, VP of Engineering at Zilliz. “But teams buy on total cost and real production behavior. We built VDBBench as an open, reproducible benchmark so any team can see what a vector database will actually cost and how it will behave on their own workload — and decide for themselves.”

Article content

What’s New in VDBBench

Article content

Building on VDBBench 1.0, which moved benchmarking closer to production by testing under streaming ingestion, filtering, recall, and concurrency, the latest release adds cost as a first-class dimension through four new cloud-oriented test cases:

Article content

Article content

The results feed a new Cost Leaderboard with a cost-performance (Pareto) view that models operating cost at target QPS levels, showing where usage-metered serverless pricing is most efficient at low, spiky volume and where provisioned, flat-rate capacity becomes more cost-effective for the sustained traffic typical of production workloads.

Article content

Built to Benchmark Any Vector Database

Article content

Article content

is open-source and supports more than 30 vector databases and search systems. To show the new cases in action, the Cost Leaderboard launches with a sample evaluation of several widely used managed vector databases — Pinecone, Turbopuffer, and Zilliz Cloud — chosen to illustrate how differently products can behave on freshness, filtering, multitenancy, and cold starts. The comparison is a demonstration, not a ranking: because the benchmark is open, teams can reproduce every case, swap in any other candidate, and run it against their own production-like data.

Article content

Article content

In the sample run, for instance, Zilliz Cloud returned newly inserted data as searchable immediately at a single-digit-dollar bulk-load cost, held a near-flat cold-start latency profile, and grew more cost-effective as sustained query volume rose — an illustration of the performance-and-cost balance the new cases are built to reveal. A product tuned for a different workload could show different strengths, which is exactly why every case is open to reproduce.

Article content

“This isn’t about naming a winner,” Luan added. “Different databases are built for different workloads, and the right choice depends on freshness, filtering, tenancy, budget, and many other factors. The results aren’t ours to declare — they’re yours to reproduce.”

Article content

Availability

Article content

The VDBBench Cost Leaderboard is available now at zilliz.com/vdbbench-leaderboard-v2. VectorDBBench is open source on GitHub, where teams can reproduce the cases or benchmark their own candidates, and share results via GitHub or the Milvus Discord community.

Article content

Article content

About Zilliz

Article content

Zilliz is a leading AI data infrastructure company and the creator of Milvus, the world’s most widely adopted open-source vector database, with 45,000+ GitHub stars and over 100 million Docker pulls. Zilliz helps enterprises and AI startups make their unstructured data searchable, analyzable, and governable — turning text, images, audio, video, and more into a strategic asset for production AI.

Article content

Zilliz’s technology centers on Milvus and Zilliz Cloud. Milvus is an open-source, lake-native vector database purpose-built for 100-billion-scale vector search. Zilliz Cloud extends that foundation into a fully managed Vector Lakebase platform, combining the high-throughput, low-latency serving capabilities of vector databases with the openness, scalability, and economics of multimodal data lakes. Zilliz powers more than 10,000 enterprises and AI-native startups worldwide, including MiniMax, OpenEvidence, Filevine, Exa, Salesforce, and Read AI.

Article content

Headquartered in Redwood Shores, California, Zilliz is backed by leading investors, including Aramco’s Prosperity 7 Ventures, Temasek’s Pavilion Capital, Hillhouse Capital, 5Y Capital, Yunqi Partners, and Trustbridge Partners. Learn more at Zilliz.com.

Article content

Article content

Article content

Article content

View source version on businesswire.com:

Article content

Article content

Article content

Contacts

Article content

Media Contact:

Article content

Article content

Molly Chen

Article content

Article content

Article content