Architecture

                  ╔═turbopuffer═════════════════════════════════╗ 
╔════════╗        ║  ┏━━━━━━━━━┓   ┏━━━━━━━━━━━┓   ┏━━━━━━━━━━┓ ║░
║        ║░       ║  ┃         ┃   ┃   Cache   ┃   ┃ Object   ┃ ║░
║ client ║◀──API──╬─▶┃ ./tpuf  ┃◀─▶┃ (SSD/RAM) ┃◀─▶┃ Storage  ┃ ║░
║        ║░       ║  ┃         ┃   ┃           ┃   ┃ (GCS)    ┃ ║░
╚════════╝░       ║  ┗━━━━━━━━━┛   ┗━━━━━━━━━━━┛   ┗━━━━━━━━━━┛ ║░
 ░░░░░░░░░░       ╚═════════════════════════════════════════════╝░
                   ░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░

api.turbopuffer.com routes to a cluster of Rust binaries that access your database on object storage. By default, we route to central USA, however we have other available regions.

After the first query, the namespace's vectors are cached for hours on an NVME SSD. Subsequent queries are routed to the same query node for cache locality, but any query node can serve queries from any namespace. The first query to a namespace reads object storage directly and is slow (~500ms for 1M vectors), but subsequent, cached queries to that node are faster (~35ms p50 for ~1M vectors).

turbopuffer is a multi-tenant service, which means each ./tpuf binary handles requests for multiple tenants. This keeps costs low. Enterprise customers can be isolated on request.

Each namespace has its own prefix on object storage. turbopuffer uses a write-ahead log (WAL) to ensure consistency. Every write adds a new file to the WAL directory inside the namespace's prefix. If a write returns successfully, data is guaranteed to be durably written to object storage. This means high write throughput (~10,000+ vectors/sec), at the cost of high write latency (~200ms+).

This prefix-per-namespace design makes turbopuffer highly suitable for multi-tenancy, scaling to billions of namespaces, with trillions of vectors.

Writes occur in windows of 100ms, i.e. if you issue concurrent writes to the same namespace within 100ms, they will be batched into one WAL entry. Each namespace can currently write 1 WAL entry per second. If a new batch is started within one second of the previous one, it will take up to 1 second to commit.


                               User Write            
                                 ┌─────┐             
             WAL                 │█████│             
   /{org_id}/{namespace}/wal     │█████│             
                                 └──┬──┘             
                                    │                
                                    │~200ms          
╔═══════════════════════════════════╬══════════════╗ 
║┌─────┐ ┌─────┐ ┌─────┐ ┌─────┐ ┌ ─│─ ┐           ║░
║│█████│ │█████│ │█████│ │█████│    ▼              ║░
║│█████│ │█████│ │█████│ │█████│ │     │           ║░
║└─────┘ └─────┘ └─────┘ └─────┘  ─ ─ ─            ║░
║  001     002     003     004     005             ║░
╚══════════════════════════════════════════════════╝░
 ░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░

After data is committed to the log, it is asynchronously indexed to enable efficient retrieval (■). Any data that has not yet been indexed is still available to search (◈), with a slower exhaustive search of recent data in the log.

turbopuffer currently provides strong consistency, i.e. if you perform a write, a subsequent query will immediately see the write. In the future we may allow eventual consistency for lower warm latency.

The approximate nearest neighbour (ANN) index has been optimized for object storage, to provide good cold latency (~500ms on 1M vectors). Additionally we build indexes for filters.


                          ╔══turpuffer region═══════╗          ╔═══Object Storage═════════════════╗ 
                          ║      ┌────────────────┬─╬───┐      ║ ┏━━Indexing Queue━━━━━━━━━━━━━━┓ ║░
                          ║      │ ./tpuf indexer │ ║░  │      ║ ┃■■■■■■■■■                     ┃ ║░
                          ║      └────────────────┘ ║░  │      ║ ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛ ║░
                          ║      ┌────────────────┐ ║░  │      ║ ┏━/{org_id}/{namespace}━━━━━━━━┓ ║░
                          ║      │ ./tpuf indexer │ ║░  │      ║ ┃ ┏━/wal━━━━━━━━━━━━━━━━━━━━━┓ ┃ ║░
                          ║      └────────────────┘ ║░  │      ║ ┃ ┃■■■■■■■■■■■■■■■◈◈◈◈       ┃ ┃ ║░
                          ║                         ║░  │      ║ ┃ ┗━━━━━━━━━━━━━━━━━━━━━━━━━━┛ ┃ ║░
                          ║                         ║░  ├─────▶║ ┃ ┏━/index━━━━━━━━━━━━━━━━━━━┓ ┃ ║░
                          ║      ┌────────────────┐ ║░  │      ║ ┃ ┃■■■■■■■■■■■■■■■           ┃ ┃ ║░
                          ║   ┌─▶│  ./tpuf query  │ ║░  │      ║ ┃ ┗━━━━━━━━━━━━━━━━━━━━━━━━━━┛ ┃ ║░
                       ┌──╩─┐ │  └────────────────┘ ║░  │      ║ ┃ ┏━━━━━━━━━━━━━━━━┓           ┃ ║░
╔══════════╗           │    │ │  ┌────────────────┐ ║░  │      ║ ┃ ┃Namespace Config┃           ┃ ║░
║  Client  ║────API───▶│ LB │─┼─▶│  ./tpuf query  │─╬───┘      ║ ┃ ┗━━━━━━━━━━━━━━━━┛           ┃ ║░
╚══════════╝░          │    │ │  └────────────────┘ ║░         ║ ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛ ║░
 ░░░░░░░░░░░░          └──╦─┘ │  ┌────────────────┐ ║░         ╚══════════════════════════════════╝░
                          ║   └─▶│  ./tpuf query  │ ║░          ░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░
                          ║      └────────────────┘ ║░                                              
                          ║                         ║░                                              
                          ╚═════════════════════════╝░                                              
                           ░░░░░░░░░░░░░░░░░░░░░░░░░░░                                              

The index is based on SPANN. SPANN is a centroid-based approximate nearest neighbour index. It has a fast index for locating the nearest centroids to the query vector. A centroid-based index works well for object storage as it minimizes roundtrips and write-amplification, compared to graph-based indexes like HNSW or DiskANN.

On a cold query, the centroid index is downloaded from object storage. Once the closest centroids are located, we simply fetch each cluster's offset in one, massive roundtrip to object storage.


 ╔═/{org_id}/{namespace}/index════════════════════╗ 
║┏━centroids.bin━━━━┓             ┌────────────┐ ║░
║┃■■■■■■■■■■■■■■■■■■┃             │Index Config│ ║░
║┗━━━━━━━━━━━━━━━━━━┛             └────────────┘ ║░
║┏━clusters-1.bin━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ║░
║┃┌──────────┐┌──────┐┌───┐┌────────────┐┌────┐┃ ║░
║┃│■■■■■■■■■■││■■■■■■││■■■││■■■■■■■■■■■■││■■■■│┃ ║░
║┃└──────────┘└──────┘└───┘└────────────┘└────┘┃ ║░
║┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛ ║░
║┏━clusters-2.bin━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ║░
║┃┌───────┐┌────────────┐┌───┐┌────────┐┌─────┐┃ ║░
║┃│■■■■■■■││■■■■■■■■■■■■││■■■││■■■■■■■■││■■■■■│┃ ║░
║┃└───────┘└────────────┘└───┘└────────┘└─────┘┃ ║░
║┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛ ║░
╚════════════════════════════════════════════════╝░
 ░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░

In reality, there are more roundtrips required for turbopuffer to support consistent writes. From first principles, each roundtrip to object storage takes ~30-50ms. Thus five roundtrips for a cold query should take as little as ~250ms. However, today cold queries are ~500ms due to various inefficiencies. We're working on closing this gap.

When the namespace is cached in NVME/memory rather than fetched directly from object storage (~500ms), the query time drops dramatically to ~35ms p50. The majority of the warm query time is spent on a single roundtrip to object storage for consistency, which we can relax on request for sub 10ms queries.


                                                                           
             │               │             │              │              
 ┌──────────┐ ┌─────────────┐ ┌───────────┐                              
 │Namespace │││Filter Index │││  Filter   ││              │              
 │  Config  │ │   Headers   │ │   Index   │                              
 └──────────┘│└─────────────┘│└───────────┘│┌────────────┐│   ┌─────┐    
                                            │  Clusters  │    │ WAL │    
 ┌──────────┐│┌─────────────┐│┌───────────┐│└────────────┘│   └─────┘    
 │  Index   │ │  Centroid   │ │  Cluster  │                              
 │  Config  │││    Index    │││  Headers  ││              │              
 └──────────┘ └─────────────┘ └───────────┘                              
 ────────────┴───────────────┴─────────────┴──────────────┴─────────────▶
  Roundtrip 1   Roundtrip 2    Roundtrip 3   Roundtrip 4     Roundtrip 5

Contact
Email us
© 2024 turbopuffer Inc.
Privacy PolicyTerms of service
SOC2 Type 1 certified