Control Plane Operations

turbopuffer on-prem includes our control plane agent which we can use to propose adjustments to your cluster, subject to your approval. See our documentation on the operational model of our control plane agent.

When turbopuffer proposes a modification to your cluster, the agent will create an associated Kubernetes resource for this operation:

$ kubectl get turbopufferoperations -n turbopuffer --sort-by=.metadata.creationTimestamp
NAME               APPROVED   STATE               AGE     DETAILS
qjh4uts8557qxly3   true       SUCCESS             7m58s   Upgrade to 46bea73f769ed2e4...
jsilyy4wu8skamb9   true       SUCCESS             7m57s   Upgrade to c7ce48b6be870806...
kqc3ltskmsx32nr7   true       SUCCESS             7m46s   Upgrade to 60b44ccd78eaf20c...
pnblpzmh5gp5fwsy   true       SUCCESS             7m46s   Upgrade to 32c171415362b200...
0a8n5w41jyj8i35v   false      REQUIRES_APPROVAL   7m46s   Restart pods: turbopuffer-query-0*

You can approve an operation by editing the Kubernetes configuration, for example, we approve 0a8n5w41jyj8i35v by running:

kubectl patch turbopufferoperation -n turbopuffer 0a8n5w41jyj8i35v --type=merge -p '{"spec": {"approved": true}}'

These operations are typically logged by your Kubernetes cluster, and can be audit logged in the way you already do for your cluster.

Automatic approval of operations

By default our cluster agent requires manual approval from a human to perform any operation, however for certain routine maintenance tasks we suggest allowing auto approval. Specifically for the following operations:

• Restart pods: Allows us to rapidly restart and pods we see behaving problematically via telemetrics.
• Trigger reindex: Allows us to force a reindex of a namespace.
• Trigger compaction: Triggers a compaction of the LSM for a given namespace, which can improve query performance.
• Trigger gc: Trigger a manual run of our garbage collector, which will reduce object storage usage.