Sizing and Capacity Planning
turbopuffer gets faster the more of your data it keeps in cache. Query nodes hold namespaces on NVMe SSD and in memory, in front of object storage. Add query nodes and more of your working set stays cached, so latency drops. Run fewer and you spend less, while queries to evicted data read object storage and take longer. How much to spend is your call. The turbopuffer team can recommend a starting size and help you tune it from telemetry, and you can scale up or down at any time.
A BYOC cluster runs two kinds of turbopuffer nodes:
- Query nodes serve queries and writes from an NVMe SSD and memory cache in front of object storage.
- Index nodes run background jobs: indexing, WAL compaction, and other maintenance.
Each query and index pod gets a dedicated node, so the pod count is the node
count. Router pods run on query nodes, and their count follows
min_query_nodes. Node pools autoscale with Karpenter on AWS and Azure and the
cluster autoscaler on GCP, so nodes are created and removed as pods are
scheduled. You choose the pod counts.
Index nodes autoscale on their own, while query nodes stay at the count you set.
Minimums and maximums
Your BYOC kit sets the node floors in turbopuffer.tfvars, and terraform apply
writes them to values.yaml as turbopuffer.min_replicas:
min_query_nodes = 6
min_index_nodes = 1
There is no single right minimum. For most clusters, min_index_nodes = 1 is
enough and the autoscaler adds index nodes when needed. Raise it only if you
want capacity guaranteed rather than waiting for the cloud provider to supply
nodes.
The maximum number of index nodes comes from the node pool limits in Terraform
(10 per zone on AWS and Azure, 128 on GCP). Don't raise max_replicas in Helm alone, since pods beyond
the node pool limit can't be scheduled.
Check that your cloud quota covers the maximum node count for both pools, plus any pinned nodes.
Query nodes
There is no autoscaler for query nodes, since adding and removing query nodes
moves namespaces between caches and causes cache thrashing. The query
StatefulSet stays at the size you set, and upgrades keep its current size.
Raising min_query_nodes scales up on the next deploy. The turbopuffer team can also scale query nodes through the
control plane.
Each namespace is routed to one query node so that it stays in that node's cache. Size query nodes so that:
- The working set fits in cache. The combined NVMe cache of all query nodes should hold the namespaces you query regularly, with headroom for growth. If it doesn't, namespaces are evicted and the next query to them reads object storage and is slow.
- CPU has headroom at peak. Plan for peak QPS, not average QPS.
- Hot namespaces have room. A single namespace with very high QPS is limited
by one node. Spread it across more nodes with
ingress.read_replicas, or give it dedicated nodes with pinning.
Add query nodes when the following widgets on the BYOC Datadog dashboard trend the wrong way:
| Widget | What to look for |
|---|---|
| NVMe cache age (query) | Age dropping over time. Data is evicted sooner, so the cache is too small for the working set. |
| Memory cache max evicted age (query) | Same signal for the memory cache. |
| Storage cache hit ratio (query pods) | Ratio dropping, so more queries read object storage. |
| Total CPU usage by query host (%) | Hosts close to saturation at peak. A single hot host points to a hot namespace rather than too few nodes. |
| Query latency over time | Rising p90 and p99. |
| tpuf 408,429,499 responses (queries) | 429s and 408s as query concurrency or timeouts are hit. |
| Query pods | Ready and not ready replicas, to confirm a scale-up has finished. |
Index nodes
Index nodes autoscale with a Kubernetes HPA on the turbopuffer-index-0
Deployment. The HPA scales on queue delay: the age of the oldest job waiting
to be picked up, with a target of about 4 minutes. If the oldest job has waited
longer than the target, the HPA adds nodes in proportion. When the queue is empty,
it scales down to the number of nodes needed for jobs that are still running,
and never below min_index_nodes. Index nodes don't scale to 0, since
compaction, garbage collection, and recall jobs also run on them.
Scale-up adds at most 1 node per minute. Scale-down waits 3 minutes and then removes at most 3 nodes every 2 minutes. The autoscaler handles bursts such as backfills, but because it adds about 1 node per minute a large backfill takes a while to reach full capacity.
If desired index pods rarely rise above min_index_nodes, the floor is higher
than your workload needs and you can lower it.
Data that hasn't been indexed yet is still searched, more slowly. If a namespace
falls more than 2 GB or 1 million rows behind, writes to it return HTTP 429
until indexing catches up (see
disable_backpressure). Check:
| Widget | What to look for |
|---|---|
| Indexing Job Max Queue Time | Jobs waiting well past ~4 minutes for long periods. |
| Unindexed bytes | Unindexed data per namespace that grows instead of draining. |
| Indexing autoscaling threshold | Values above 1 mean the HPA wants more index nodes. |
| Index pods | Desired and ready replicas. Desired replicas flat at the maximum mean the limit is too low, and desired replicas that never rise above the minimum mean the floor is too high. |
| Insufficient Capacity Errors from Cloud | The cloud provider can't supply nodes of the requested type in your zones. |
| Total CPU usage by index host (%) | Index node utilization. |
| Jobs waiting and Jobs running | Queue depth and running jobs by queue. |
