Control Plane

turbopuffer on-prem includes our control plane agent which we can use to propose adjustments to your cluster, subject to your approval. You may also allow us to automatically apply certain operations, allowing for a higher SLA in your on-prem cluster. See our documentation on the operational model of our control plane agent.

Currently supported operations

The control plane agent is source-available upon request, and supports a limited amount of hardcoded operations to operate your cluster. The currently supported operations are listed below:

• Restart pods: Allows us to rapidly restart and pods we see behaving problematically via telemetrics.
• Scale Pods: Allows us to change the desired replica count for your index and query nodes.
• Trigger reindex: Allows us to force a reindex of a namespace.
• Trigger compaction: Triggers a compaction of the LSM for a given namespace, which can improve query performance.
• Trigger gc: Trigger a manual run of our garbage collector, which will reduce object storage usage.
• Upgrade tpuf: Updates the image digest for your turbopuffer deployment triggering a rollout for the new version of turbopuffer.

We typically find our onprem customers operate best allowing us to perform routine maintenance operations (restarting pods, triggering indexing/compaction/gc). These operations may need to be performed unpredictably to react to shifting workloads or recent upgrades to your turbopuffer deployment.

Configuring auto-approval for your cluster

By default all control plane operations require manual approval, but if you would like to make them automatic, you can edit the tpuf-ctl-cluster-controller-config-* ConfigMap found in the tpuf-ctl-cluster namespace of your cluster. This ConfigMap defines a controller_config.yaml file with the following syntax:

apiVersion: v1
data:
  controller_config.yaml: |-
    vendor_url: https://tpuf-ctl-vendor.vercel.app/heartbeat/gcp-us-central1
    approvals:
      restart: manual # can set to: auto
      scale: manual # can set to: auto
      reindex: manual # can set to: auto
      compact: manual # can set to: auto
      gc: manual # can set to: auto
      upgrade: manual # can set to: auto
kind: ConfigMap

Viewing proposed operations in your cluster

When turbopuffer proposes a modification to your cluster, the agent will create an associated Kubernetes resource for this operation:

$ kubectl get turbopufferoperations -n turbopuffer --sort-by=.metadata.creationTimestamp
NAME               APPROVED   STATE               AGE     DETAILS
qjh4uts8557qxly3   true       SUCCESS             7m58s   Upgrade to 46bea73f769ed2e4...
jsilyy4wu8skamb9   true       SUCCESS             7m57s   Upgrade to c7ce48b6be870806...
kqc3ltskmsx32nr7   true       SUCCESS             7m46s   Upgrade to 60b44ccd78eaf20c...
pnblpzmh5gp5fwsy   true       SUCCESS             7m46s   Upgrade to 32c171415362b200...
0a8n5w41jyj8i35v   false      REQUIRES_APPROVAL   7m46s   Restart pods: turbopuffer-query-0*

You can approve an operation by editing the Kubernetes configuration, for example, we approve 0a8n5w41jyj8i35v by running:

kubectl patch turbopufferoperation -n turbopuffer 0a8n5w41jyj8i35v --type=merge -p '{"spec": {"approved": true}}'

Custom Resource Definition transitions are generally logged by your cloud provider's Kubernetes cluster, so your standard audit logging tools should support turbopuffer's operations.