turbopuffer v3

turbopuffer v3 is now rolling out to production. We are enabling v3 namespace by namespace on multi-tenant clusters and monitoring performance before rolling it out everywhere. If you want to enable v3 in your namespaces immediately, please contact us.

Benchmark

turbopuffer v3 fundamentally redesigns the tpuf engine to make search faster in every respect — including for text, vector, regex, aggregation, and filtered searches. The chart below shows our progress optimizing v3 on a set of core benchmarks. As of Nov 16, all v3 benchmarks now meet or beat v2.

×

Benchmarks forthcoming

Date

BenchmarkSpeedup
Count, grouped
hot
50 x
Order by attribute, filtered
hot
9.91 x
Count
cold
2.93 x
Full-text search
cold
1.19 x
Vector search
hot
1.15 x
Count, filtered
cold
1.08 x
Count
hot
1.07 x
Count, filtered
hot
1.07 x
Full-text search
hot
1.06 x
Vector search
cold*
1.05 x
Order by attribute
hot
1.05 x
Benchmark methodology

Each point is how many times faster v3 is than production: v2 p90 divided by v3 p90. The v2 baseline is that night's p90, unless v3 was slower that night, in which case it is the median v2 p90 across every night shown here. Above 1x, v3 is faster.

Test namespaces live in gcp-us-central1 and hold 10 million documents. A c4a-highmem-32 in us-central1-c sends 8 queries per second for 10 minutes. Hot runs wait until the cache reaches a 100% hit ratio. Cold runs send queries with the cache disabled.

Vector workloads use 1024-dimensional Cohere embed-multilingual-v3 embeddings from the Cohere Wikipedia dataset. Full-text workloads use MS MARCO, with 1024-dimensional Cohere embed-english-v3 embeddings for the vector query. Attribute ordering uses synthetic documents with a numeric popularity attribute and 100 category values.

*cold query latency is inherently noisy; we do not expect v3 to significantly reduce latency for cold vector queries

Dev Log

  1. As of today, every query plan on the v3 benchmark is as fast as or faster than v2 at p90.

    The benchmark covers a small handful of query plans that we have used to guide our development. Production traffic is obviously much more varied. We have been shadowing production queries on v3 to find unoptimized query shapes not covered by the benchmarks, and fixing any regressions they turn up.

    turbopuffer v3 is now rolling out to production. Over the next few weeks, we'll be enabling v3 namespace-by-namespace on multi-tenant clusters and monitoring performance. Once v3 serves stable production traffic in those regions without surprises, we plan to roll out across all production traffic, including single-tenant and BYOC clusters.

  2. Full-text search is now faster across the board on turbopuffer v3.

    FTS v2 already used a highly tuned, vectorized query engine based on MAXSCORE. We ported v2's query engine to run on the new v3 storage layout, then tuned it with a series of small changes until performance beat v2.

    Unlike v2, which runs this query engine on a bespoke storage layout built specifically for full-text search, v3 runs the same engine on an efficient, CPU-friendly storage layout shared by every query plan.

  3. p90 latency for grouped counts on the v3 benchmark dropped by 65%. The query counts 10 million documents, grouped by a string attribute with 1,000 unique values. When we started tracking it, v3 was already 15x faster than v2, largely thanks to columnar storage, before we had tuned the aggregation code. Now v3 is about 50x faster than v2 for grouped aggregations.

    Aggregations haven't historically been a large workload on turbopuffer, so the code behind them was largely unoptimized. We are being pushed to support more aggregation query plans at much greater scale, and the v3 architecture gives us the foundation to do it well.

    Before this change, we aggregated one row at a time. For each row, we copied its group key (the value of the attribute we group by), searched a tree map for that key, then updated every aggregate for that group before moving on to the next row. Counting itself is quite cheap, so the copy and the search dominated the cost.

    Now we group with a hash map and only copy a group key the first time we see it, rather than on every row. We also vectorize aggregation by processing each page of rows in a tight loop.

  4. The first search engine with fully vectorized query executionVectorized execution is the foundation of turbopuffer v3's query engine. This will allow turbopuffer to provide strong performance on many more types of queries.Adrien Grand (Engineer)
  5. On 10/09, we started tracking aggregation queries on the v3 benchmark. A count over 10 million docs that groups by one attribute is already 15x faster on v3 than v2, even though we haven't yet tuned the aggregation code. The speedup comes in large part from v3's columnar storage engine, which lets us read values in bulk instead of one row at a time.

    Ungrouped counts are already fast in production (25ms p90 hot). Unfiltered counts just read index statistics, and most filtered counts can popcount a bitmap of matching documents.

    For unfiltered grouped counts, we need the group-by attribute value (or values, for grouped counts over multiple attributes) of every document in the namespace. v2 fetches values with a separate lookup per document and attribute, so the benchmark query does 10 million separate lookups. v3 stores each attribute as a contiguous column, so it reads the values in bulk.

  6. RIP, vector databaseturbopuffer is pushing the frontier of search. To do that, we have to fundamentally redesign our storage architecture so the vector index is no longer primary.Dan Harrison (Engineer)
  7. turbopuffer v3 uses a columnar storage engine. Data is stored physically in a struct-of-arrays layout, grouping same-typed values contiguously, so scans take full advantage of available {memory, SSD} bandwidth and can be more easily vectorized using SIMD instructions.

    Up until this point, tpuf v3 stored columnar data densely: as a flat array of values (Vec<T>). This was a simple approach and gave us the right foundation for building optimal compressed forms. Now, we've enabled various compressed columnar encodings. For example, if all the values in a column happen to be identical (this happens more than you'd think!), we can collapse it into a Constant<T> encoding, storing the value once with a length: { value: T, len: u32 }. Or, for columns with few unique strings, we can use a Dictionary<T> encoding to store each string once in a table, with each row storing just an index into the table: { indexes: Column<u32>, values: Column<T> }. We can also collapse consecutive runs into a single value and its end position (Run-Length encoding), or store a column's most common value once instead of per row (Frequency encoding).

    Compressed columnar encodings not only reduce the bytes we need to read, they also enable "compute-on-compressed": evaluating a predicate (age = 25) on a Constant<T> column of any length is O(1), since all values either match or don't based on a single comparison. Fewer bytes read and less compute per query means we can make better use of the available bandwidth and serve more queries per second.

    I'm writing a more detailed blog post on this topic, stay tuned!

    Morgan Gallant (Engineer)
  8. On 9/19, we saw p90 latency for hot and cold FTS queries on tpuf v3 drop by up to 80%. Building on our past work that batched reads of postings, we now retain these batches through query execution. Previously, we converted these batches into individual rows for simplicity and correctness in our initial implementation. In this change, we operate on and score batches of postings directly, in a very similar manner to how our current FTS v2 query engine vectorizes scoring.

    Cold full-text search queries saw a compounding improvement because we also eliminated an object storage round trip on cold FTS reads. To understand how we improved cold latency for FTS queries, we must understand our data dependencies. At a high level, to score documents for a BM25 query, we must load text postings for the query terms, any attribute data needed for filtering, and BM25 statistics for the corpus (e.g. document frequencies) from object storage. Previously, tpuf v3 made these requests serially, loading corpus statistics during planning and postings/attributes for filtering at execution time. A large part of optimizing tpuf cold workloads comes from reducing serial round trips to object storage. Here, we've pushed loading corpus statistics down to be fetched concurrently with the remainder of our postings and attribute data, just before scoring documents. By combining these into a single round trip, we shave hundreds of milliseconds off cold FTS query latency.

    Matt Cuento (Engineer)
  9. On 09/19, we saw a 95% reduction in the runtime for unfiltered order-by queries. Order-by queries have two halves: scanning over documents and accumulating the top-k set. Depending on the specific query, either or both can be the slow part. When working with unfiltered order-by queries, the bottleneck typically falls on the top-k selection, since we must consider each row for inclusion.

    To make queries an order of magnitude faster, it isn't enough to micro-optimize our top-k operation, we must also fundamentally reduce the number of rows we consider for inclusion. We achieve this by adding a dynamic filter to queries to rapidly exclude rows that could never enter our result set, leveraging our vectorized filter evaluation for fast processing.

    Consider SELECT _ ORDER BY age asc LIMIT 2 run over the table [1, 10, 4, 8, 5, 15, 7]. We add a < DYN filter to our query. This dynamic filter is initially inactive, but once we've scanned the first two rows, we have a candidate result set: [1, 10]. From this we know any new candidate must have age < 10 to make the result set, so we can skip the 6th row (age = 15). After the third row, the result set is [1, 4] and we can skip evaluating all other rows. By updating the dynamic filter after each row, we progressively skip more and more rows in bulk, making the query much faster. This skipping mechanism is quite similar to how our MAXSCORE algorithm executes full-text queries, but applied on order-by queries.

    Xavier Denis (Engineer)
  10. We designed the tpuf v3 query engine to benefit from vectorization everywhere. Our last two devlogs described how we batched reads from storage, but vectorization also extends beyond storage and into the query engine. The first part of the engine to benefit from this is our filter evaluation logic.

    As we scan over the documents in a namespace to execute a query, we must check that each candidate row satisfies all filters. We do this by building a bitmap, which is an intersect (AND) or union (OR) of the bitmaps representing which documents match each filter. Previously we filled this bitmap one document at a time, now we build it a page at a time.

    Vectorization in tpuf v3 is more than just asking the storage to read and return a range of rows, then operating over them in batches. We also perform "compute pushdown": instead of asking the query engine to evaluate the filter on each row, we ask the underlying storage to answer questions like "are all these rows = 8?". By structuring things this way, we can leverage the specific physical encodings of the data to answer questions in bulk and maximize throughput. Compute pushdown is fundamental to the performance of v3, and a big part of the reason v3 filtering is now faster than current production (v2). We'll talk in more detail about pushdown in the upcoming blog post on our columnar storage layout.

    Xavier Denis (Engineer)
  11. After the large speedup we observed implementing batched reads for FTS, we applied a similar change to our order_by query plans, allowing them to take advantage of our new columnar storage layout, and avoiding the overhead of materializing each individual row during read. The majority of optimizations coming to our query engine — spanning filter execution, order_by, full-text search, and aggregation queries — will have roughly of this shape, e.g. taking advantage of the new physical layouts to find opportunities for batched execution and vectorization.

    We're working on a deeper blog post discussing our new columnar layouts and the impact it will have on our query plans. We expect to publish it in a couple of weeks.

    Also of note, we simplified how we execute filters for full-text queries, temporarily causing the performance regression you observe in the benchmark chart. The naive implementation evalutes the filter on every row prior to scoring, which is inefficient, but the codepath sets the foundation for significant optimizations on filtered full-text search queries. In particular, we will be able to incorporate filter execution directly into our vectorized full-text query engine, like we do today in production, and skip filtering unnecessary rows.

    Morgan Gallant (Engineer)
  12. On 09/08, we implemented batched reads in the v3 full-text engine, leading to a ~9x performance improvement. Sometimes, optimization is not about being clever, but simply avoiding mistakes. Postings are stored as columns, and we were reading them a row at a time. The fix here was obvious, we batched reads from storage. Notably, we're not even batching execution during scoring yet, that will come soon.

    Modern CPUs are happiest when their pipelines are full, and batching helps keep them full. Full-text search on tpuf v2 took advantage of this with a bespoke storage system and query engine to enable batching all the way through the pipeline, from reads to scoring.

    Unlike v2, v3 unifies every index that uses posting lists on the same internal storage format, so this fix benefits all of them, not just full-text.

    We're working on a deeper technical post about this, and we'll share it in the next few weeks.

    Xavier Denis (Engineer)

follow for updates