Control Plane

turbopuffer BYOC includes our control plane agent which we can use to apply adjustments to your cluster. You may also choose to manually approve certain operations, though doing so will limit our team's ability to manage your cluster and respond to incidents. See our documentation on the operational model of our control plane agent.

Currently supported operations

The control plane agent is source-available upon request, and supports a limited amount of hardcoded operations to operate your cluster. The currently supported operations are listed below:

• Restart pods: Allows us to rapidly restart and pods we see behaving problematically via telemetrics.
• Scale Pods: Allows us to change the desired replica count for your index and query nodes.
• Trigger reindex: Allows us to force a reindex of a namespace.
• Trigger compaction: Triggers a compaction of the LSM for a given namespace, which can improve query performance.
• Trigger gc: Trigger a manual run of our garbage collector, which will reduce object storage usage.
• Upgrade tpuf: Updates the image digest for your turbopuffer deployment triggering a rollout for the new version of turbopuffer.

We typically find our BYOC customers operate best allowing us to perform routine maintenance operations (restarting pods, triggering indexing/compaction/gc). These operations may need to be performed unpredictably to react to shifting workloads or recent upgrades to your turbopuffer deployment.

Configuring manual approval for your cluster

By default all control plane operations are automatically approved, but if you would like to make one or more operations require manual approval, you can edit the tpuf-ctl-cluster-controller-config-* ConfigMap found in the tpuf-ctl-cluster namespace of your cluster. This ConfigMap defines a controller_config.yaml file with the following syntax:

apiVersion: v1
data:
  controller_config.yaml: |-
    vendor_url: https://tpuf-ctl-vendor.vercel.app/heartbeat/gcp-us-central1
    approvals:
      upgrade: auto  # can set to `manual`, but weakly discouraged

      restart: auto  # can set to `manual`, but strongly discouraged
      scale:   auto  # can set to `manual`, but strongly discouraged
      reindex: auto  # can set to `manual`, but strongly discouraged
      compact: auto  # can set to `manual`, but strongly discouraged
      gc:      auto  # can set to `manual`, but strongly discouraged
kind: ConfigMap

Viewing proposed operations in your cluster

When turbopuffer proposes a modification to your cluster, the agent will create an associated Kubernetes resource for this operation:

$ kubectl get turbopufferoperations -n turbopuffer --sort-by=.metadata.creationTimestamp
NAME               APPROVED   STATE               AGE     DETAILS
qjh4uts8557qxly3   true       SUCCESS             7m58s   Upgrade to 46bea73f769ed2e4...
jsilyy4wu8skamb9   true       SUCCESS             7m57s   Upgrade to c7ce48b6be870806...
kqc3ltskmsx32nr7   true       SUCCESS             7m46s   Upgrade to 60b44ccd78eaf20c...
pnblpzmh5gp5fwsy   true       SUCCESS             7m46s   Upgrade to 32c171415362b200...
0a8n5w41jyj8i35v   false      REQUIRES_APPROVAL   7m46s   Restart pods: turbopuffer-query-0*

You can approve an operation by editing the Kubernetes configuration, for example, we approve 0a8n5w41jyj8i35v by running:

kubectl patch turbopufferoperation -n turbopuffer 0a8n5w41jyj8i35v --type=merge -p '{"spec": {"approved": true}}'

Custom Resource Definition transitions are generally logged by your cloud provider's Kubernetes cluster, so your standard audit logging tools should support turbopuffer's operations.