# Standard BYOC Setup This guide takes you from an empty cloud account to a running [Standard BYOC](/docs/byoc#standard-byoc) cluster. Work through the runlist in order and check off each step as you go. Before you start, check the [requirements](/docs/byoc/requirements) and [size your cluster](/docs/byoc/sizing). For each cluster, turbopuffer provides a BYOC kit with everything the runlist needs. If you get stuck on a step, ask us in your shared Slack channel. ## Your kit contents ```txt byoc-kit ├── README.md ├── aws │ ├── main.tf │ └── turbopuffer.tfvars ├── cosign.pub ├── gcp │ ├── main.tf │ └── turbopuffer.tfvars ├── azure │ ├── main.tf │ └── turbopuffer.tfvars ├── scripts │ ├── apikey.py │ ├── generate-secrets.py │ └── sanity.sh ├── values.yaml (generated) # configuration file generated by Terraform ├── values.secret.yaml (generated) # sensitive configuration file generated by Terraform ├── compute_classes.yaml (generated, GCP only) # GKE ComputeClass manifest generated by Terraform └── metrics-keys.yaml # configuration file provided by turbopuffer ``` ## Runlist 1. [ ] **Prerequisites:** Check you have everything you need: - [ ] Verify you have `terraform`, `kubectl`, and `helm` installed. - [ ] Provision a **fresh sub-account / project** for your new cluster. - [ ] Enable your cluster to pull images from our registries: **GCP:** Provide your **GCP project number** to the turbopuffer team. The kit's Terraform creates a dedicated node service account for image pulls instead of using the Compute Engine default service account, and `terraform output -raw node_service_account_email` prints its email if you prefer to allowlist only that account. **AWS:** Provide us with your **AWS account ID**. **Azure:** Provide the **multitenant application (client) ID** that [will be used for your Azure clusters](https://learn.microsoft.com/en-us/azure/container-registry/authenticate-aks-cross-tenant#pull-images-from-a-container-registry-to-an-aks-cluster-in-a-different-microsoft-entra-tenant). 2. [ ] **Cluster configuration:** Apply the Terraform configuration to set up the Kubernetes cluster and bucket. - [ ] **GCP:** `cd gcp` **AWS:** `cd aws` **Azure:** `cd azure` - [ ] Run `terraform init` to set up the required providers. - [ ] Fill in the required values in `turbopuffer.tfvars`**GCP:** . The `query_skus`, `index_skus`, and `maintenance_machine_type` variables control which GCP machine types back the query, index, and maintenance node pools. The defaults work for most deployments. - [ ] Apply the Terraform configuration: `terraform apply -var-file=turbopuffer.tfvars` - [ ] Return to the kit root: `cd ..`. The remaining steps run from there. 3. [ ] **Configure `kubectl`:** Add your new cluster context to `kubectl`. - [ ] Run **GCP:** `gcloud container clusters get-credentials CLUSTER_NAME --project PROJECT_ID --region REGION` **AWS:** `aws eks update-kubeconfig --region REGION --name CLUSTER_NAME` **Azure:** `az aks get-credentials --name=CLUSTER_NAME --resource-group=RESOURCE_GROUP` - [ ] Run `kubectl config get-contexts` and confirm the context is correct. - [ ] Run `kubectl get pods` and confirm the command succeeds (no output).**GCP:** **Apply ComputeClass manifests:** Terraform writes a `compute_classes.yaml` at the kit root that defines GKE `ComputeClass` priorities across the query and index node pools. Apply it before installing Helm so the autoscaler has the priority order in place when the first pods schedule: `kubectl apply -f compute_classes.yaml` 4. [ ] **Configure Helm:** The Terraform command will have generated a `values.yaml` file at the kit root, which contains values for Helm. Edit this file and set any other necessary values. Refer to `values.schema.json` for a description of valid configurations. - [ ] Review the `tpuf_config` values suggested by turbopuffer for your BYOC deployment. You can find information about these settings and more in our [BYOC configuration](/docs/byoc/configuration) documentation. - [ ] If you serve multiple customers from this cluster and want each one's data in its own bucket, set up a [bucket per customer](/docs/byoc/configuration#bucket-per-customer) now. Existing data is not migrated when you add one later. - [ ] If you terminate TLS with your cloud provider's managed certificates, follow [using cloud-provider-managed TLS certificates](#using-cloud-provider-managed-tls-certificates). - [ ] If you host images in your own registry, follow [using a custom registry](#using-a-custom-registry-for-your-turbopuffer-cluster). - [ ] If you deploy with ArgoCD or another GitOps tool, follow [deploying with GitOps tools](#deploying-with-gitops-tools). 5. [ ] **Generate API keys:** Run `./scripts/generate-secrets.py` to generate `values.secret.yaml`. The script will generate an org ID and API key, along with a token for intra-cluster communication. 6. [ ] **Deploy turbopuffer:** - [ ] **Log in to the Helm registry:** Run **GCP:** `helm registry login us-central1-docker.pkg.dev`. Your own credentials pull the chart; the node service account only pulls images. **AWS:** `aws ecr get-login-password --region us-west-2 | helm registry login --username AWS --password-stdin 961341552108.dkr.ecr.us-west-2.amazonaws.com` **Azure:** `az login --tenant 398cc17e-41b3-44de-929a-dc4048da9592 && az acr login --name turbopuffer` - [ ] **Install the Helm chart:** Run **GCP:** `helm install -n default turbopuffer oci://us-central1-docker.pkg.dev/turbopuffer-onprem/charts/tpuf --values=values.yaml --values=values.secret.yaml --values=metrics-keys.yaml` **AWS:** `helm install -n default turbopuffer oci://961341552108.dkr.ecr.us-west-2.amazonaws.com/turbopuffer/turbopuffer/charts/tpuf --values=values.yaml --values=values.secret.yaml --values=metrics-keys.yaml` **Azure:** `helm install -n default turbopuffer oci://turbopuffer.azurecr.io/turbopuffer/charts/tpuf --values=values.yaml --values=values.secret.yaml --values=metrics-keys.yaml` - [ ] **For subsequent updates**, run **GCP:** `helm upgrade -n default turbopuffer oci://us-central1-docker.pkg.dev/turbopuffer-onprem/charts/tpuf --values=values.yaml --values=values.secret.yaml --values=metrics-keys.yaml` **AWS:** `helm upgrade -n default turbopuffer oci://961341552108.dkr.ecr.us-west-2.amazonaws.com/turbopuffer/turbopuffer/charts/tpuf --values=values.yaml --values=values.secret.yaml --values=metrics-keys.yaml` **Azure:** `helm upgrade -n default turbopuffer oci://turbopuffer.azurecr.io/turbopuffer/charts/tpuf --values=values.yaml --values=values.secret.yaml --values=metrics-keys.yaml` - If you render the chart and apply it with your own tooling (e.g. `helm template | kubectl apply`), use `kubectl apply --server-side`. The chart includes a CRD too large for client-side apply's 256KB annotation limit. 7. [ ] **Run post-deployment sanity checks:** - [ ] `TURBOPUFFER_API_KEY= scripts/sanity.sh` will query your turbopuffer cluster directly, verifying that core operations function. It will not verify certificates and may encounter a 500 error if the nodes aren't routable yet. 8. [ ] **Get access to monitoring:** Ask the turbopuffer team for access to your cluster's Datadog dashboard. See [Monitoring](/docs/byoc/operations#monitoring). Your cluster is now running. From here, [Configuration](/docs/byoc/configuration) covers changing settings and [Operations](/docs/byoc/operations) covers day-to-day tasks like adding organizations, scaling, and upgrades. ## Using a custom registry for your turbopuffer cluster By default, turbopuffer will pull from one of several turbopuffer-managed image registries, as configured in our included Terraform. However, there are many reasons you may want to host our images in a registry you control. Our Helm chart fully supports this through the following settings: ```yaml image.registry: YOUR_REGISTRY_URL control_plane.image.registry: YOUR_REGISTRY_URL ``` We expect that registry to contain two repositories, `turbopuffer` and `tpuf-ctl-cluster`, to hold the images for turbopuffer and our control plane agent, respectively. For customers on AWS, we can configure ECR Replication to automatically push the latest images into your registry. ## Using cloud-provider-managed TLS certificates Our Helm chart supports configuring TLS termination within your cluster using either `cert-manager` or native Kubernetes APIs. Your organization may already manage certificates through your cloud provider's managed certificate offering. In that case, you will need to handle TLS termination yourself. Regardless of your cloud provider, you will want to deploy turbopuffer internally by setting: ```yaml ingress.internal: true ``` ### GCP To get started, set the following in `values.yaml` and rerun `helm upgrade ...` as described in step 6. ```yaml certificates.mode: disabled ``` Adding [Google Managed Certificates](https://cloud.google.com/kubernetes-engine/docs/how-to/managed-certs) to your GKE cluster is as simple as deploying the following Kubernetes manifest alongside your turbopuffer Helm deployment. All that is required is to insert the correct value for `YOUR_DOMAIN`. ```yaml apiVersion: cloud.google.com/v1 kind: BackendConfig metadata: name: turbopuffer-router-config namespace: turbopuffer spec: healthCheck: checkIntervalSec: 10 timeoutSec: 10 port: 8080 type: HTTP requestPath: /healthz --- apiVersion: v1 kind: Service metadata: name: turbopuffer-router-backend namespace: turbopuffer annotations: cloud.google.com/backend-config: '{"default": "turbopuffer-router-config"}' spec: ports: - appProtocol: http name: http port: 80 protocol: TCP targetPort: router-http - appProtocol: https name: https port: 443 protocol: TCP targetPort: router-https selector: turbopuffer-routing-group: router type: ClusterIP --- apiVersion: networking.gke.io/v1 kind: ManagedCertificate metadata: name: managed-cert namespace: turbopuffer spec: domains: - YOUR_DOMAIN --- apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: turbopuffer-router-ingress namespace: turbopuffer annotations: kubernetes.io/ingress.class: "gce" networking.gke.io/managed-certificates: managed-cert spec: ingressClassName: "gce" defaultBackend: service: name: turbopuffer-router-backend port: number: 80 ``` ### AWS You will need to provision your certificate externally to your cluster using the AWS console or CLI. Then update your Helm `values.yaml` with the following configuration values and rerun `helm upgrade ...` as described in step 6. ```yaml ingress: certificates: mode: aws aws: certificate_arn: YOUR_CERTIFICATE_ARN ``` ## Restricting outbound connections If your firewall blocks outbound traffic by default, allow the following destinations: + Polar Signals (CPU and Heap profiling) - `35.234.93.182` (`api.polarsignals.com`) + Control Plane (Cluster Heartbeats) - `34.8.199.114/32` + Datadog (Telemetry) - `curl -s https://ip-ranges.us5.datadoghq.com/ | jq -r '(.agents.prefixes_ipv4 + .api.prefixes_ipv4 + .apm.prefixes_ipv4 + .global.prefixes_ipv4 + .logs.prefixes_ipv4 + .orchestrator.prefixes_ipv4 + .process.prefixes_ipv4 + .["remote-configuration"].prefixes_ipv4) | unique[]'` - Note: turbopuffer reports to the `us5` Datadog site, so the ranges must come from `ip-ranges.us5.datadoghq.com`. These ranges can change over time; if your firewall supports FQDN rules, allowing `*.us5.datadoghq.com` on 443 is more durable. + Teleport (Managed BYOC only, SSH access to the jump host) - Ask the turbopuffer team for the address. ## Deploying with GitOps tools By default, our Helm chart uses [the `lookup` function](https://helm.sh/docs/chart_template_guide/functions_and_pipelines/#using-the-lookup-function) to retain the currently running version across deployments, so a `helm upgrade` preserves whatever image the control plane last rolled out. Tools that render the chart without `lookup` (e.g. ArgoCD, Terraform, or other GitOps/CI tools) reset the image to your Helm values on each sync and undo the control plane's upgrades. There are two ways to handle this. With ArgoCD, you can [keep automated upgrades](#keeping-automated-upgrades-under-argocd) by deferring the image to the control plane. With any tool, you can [roll out upgrades from your own pipeline](#configuring-external-execution-for-upgrades) instead. ### Keeping automated upgrades under ArgoCD ArgoCD renders charts with `helm template`, which doesn't support `lookup` (it always returns nil), so on each sync ArgoCD would otherwise reset the image to your Helm values and undo the control plane's upgrade. To keep automated upgrades working under ArgoCD, tell ArgoCD to defer the container image to the control plane. The control plane updates the image with a [server-side apply](https://kubernetes.io/docs/reference/using-api/server-side-apply/) under the field manager `turbopuffer-controller`. By telling ArgoCD to respect that ownership, self-heal leaves the image in place while continuing to reconcile everything else from your Helm values. Set this on your ArgoCD `Application`: ```yaml spec: ignoreDifferences: - group: apps kind: StatefulSet managedFieldsManagers: - turbopuffer-controller - group: apps kind: Deployment managedFieldsManagers: - turbopuffer-controller syncPolicy: syncOptions: - ServerSideApply=true # let our field manager and ArgoCD's coexist on the object - RespectIgnoreDifferences=true # honor the rule during sync, not just in the diff view ``` Both sync options are required. `managedFieldsManagers` scopes the rule to the fields the control plane manages on those objects (the image and its version metadata), so ArgoCD keeps reconciling the rest of each object from your Helm values. `ServerSideApply=true` lets the control plane's field manager and ArgoCD's apply to the same object without conflict, and `RespectIgnoreDifferences=true` makes sync honor the rule rather than only hiding it from the diff view. Without `RespectIgnoreDifferences=true`, ArgoCD still reverts the image on self-heal even though the diff appears clean. ### Configuring external execution for upgrades In `external_execution` mode, the turbopuffer control plane will not perform upgrades directly. Instead, when a new version is available, you'll receive an automated notification to your Slack channel with the version to deploy. You can then update the `turbopuffer.initial_tag` value in your Helm values and apply it through your deployment pipeline. To enable external execution for upgrades, you can set the following in your Helm `values.yaml`: ```yaml control_plane: additional_config: approvals: upgrade: external_execution turbopuffer: image: initial_tag: force_initial_tag: true ``` #### Retrieving the target version programmatically If you want to automate this process, we expose the target version via an API endpoint that you can query from your deployment tooling: ```bash curl -H "Authorization: Bearer " \ -H "Accept: application/json" \ "https://control.turbopuffer.com/api/clusters//target_version" ``` Replace `` with your cluster's API key and `` with your cluster identifier. The response includes both the git commit SHA and the image digest: ```json { "target_version": "", "target_digest": "sha256:" } ``` You can use this endpoint in your Terraform configurations, CI/CD pipelines, or any other automation to retrieve and apply the latest target version. For example, in Terraform: ```hcl data "http" "turbopuffer_target_version" { url = "https://control.turbopuffer.com/api/clusters/${var.cluster_id}/target_version" request_headers = { Accept = "application/json" Authorization = "Bearer ${var.turbopuffer_api_key}" } } locals { turbopuffer_version = jsondecode(data.http.turbopuffer_target_version.response_body) } # Use in your Helm release configuration: # local.turbopuffer_version.target_version -> git commit SHA # local.turbopuffer_version.target_digest -> image digest (sha256:...) ``` ## Creating additional clusters You can create additional clusters by reusing your existing BYOC kit, `metrics-keys.yaml`, and control plane API key. Follow this runlist again from a copy of the kit directory so the existing cluster's Terraform state is untouched. - Set a unique `cluster_name`, `bucket_name`, and `cluster_endpoint` in `turbopuffer.tfvars`. - Generate new turbopuffer org IDs and API keys for the new cluster with `scripts/generate-secrets.py`. - For AWS, set `enable_spot_service_linked_role = false` when deploying into the same account. The role already exists and a second `terraform apply` fails trying to create it. --- This page: [/docs/byoc/standard.md](https://turbopuffer.com/docs/byoc/standard.md) All documentation pages: [/llms.txt](https://turbopuffer.com/llms.txt) All documentation in one file: [/llms-full.txt](https://turbopuffer.com/llms-full.txt)