Common Operations

While turbopuffer strives to be as low interaction as possible, there are certain manual operations you will have to perform in your BYOC deployment.

Securely partitioning your data

turbopuffer BYOC allows you to configure multiple organizations, each with their own set of API keys which you can use to scope data access. Currently, we only support creating admin API keys, that will apply to all namespaces in their organization. For this reason, if you need to ensure data is isolated we recommend creating multiple organizations instead. If this is a limitation, we recommend you contact us on Slack.

Generating org IDs and API keys

You can generate valid org IDs and API keys using any tooling that produces cryptographically random values. The format requirements are:

Org ID

  • 24 character random string
  • Alphabet: [a-z0-9] (lowercase alphanumeric only)

API Key

  • Prefix: tpuf_
  • Followed by: 32 character random string from alphabet [a-zA-Z0-9]
  • Full format: tpuf_XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

Stored API Key Hash

  • Format: base64(sha256(full_api_key))
  • The hash is computed over the complete API key including the tpuf_ prefix

Your BYOC kit includes a generate_secrets.py script that generates these values for you.

Storing org configuration

authentication:
  allowed_api_keys_sha256:
    "your24charorgidhere1234":
      - "YourBase64EncodedSHA256HashHere+40CharactersTotal="
      - "AnotherBase64EncodedSHA256HashHere+40CharactersTot="

Adding additional organizations

To add more organizations, generate new org ID and API key pairs and add them to the configuration:

authentication:
  allowed_api_keys_sha256:
    "existingorgid123456789012":
      - "ExistingOrgKeyHash..."
      - "AnotherExistingOrgKeyHash..."
    "neworgid0987654321abcdef":
      - "NewOrgKeyHash..."
      - "AnotherNewOrgKeyHash..."

Adding a new API key to an existing organization

Each organization can have multiple API keys for key rotation, different services, or other access patterns. To add a new API key, use the apikey.py script from your BYOC kit and append the generated hash to the organization's key list:

authentication:
  allowed_api_keys_sha256:
    "existingorgid123456789012":
      - "ExistingOrgKeyHash..."
      - "NewlyAddedKeyHash..."

Bring your own bucket

You may want to store a customer's data in a bucket that the customer owns, often called bring your own bucket (BYOB). This is configured by mapping the customer's org to a dedicated bucket with blob.bucket_overrides_by_org_id.

The bucket can live in a different AWS account, GCP project, or Azure tenant than the turbopuffer cluster:

  • AWS: grant the cluster's IAM role access to the bucket, or set aws_assume_role on the bucket entry to a role in the bucket's account. The role's trust policy must allow the cluster's IAM role to assume it (sts:AssumeRole).
  • GCP: grant the cluster's service account roles/storage.objectAdmin on the bucket.
  • Azure: set azure_tenant_id on the bucket entry to authenticate against the tenant that owns the storage account. The cluster's workload identity must be registered as a multi-tenant application, with a service principal in the target tenant granted Storage Blob Data Contributor on the storage account. To use a dedicated identity instead of the cluster's default, set azure_client_id on the bucket entry. The application needs a federated credential with the same issuer, subject, and audience as the cluster's workload identity.

For example:

blob:
  bucket_overrides_by_org_id:  # org ID -> bucket ID
    "5x8olkguh1l2jvtjrpgnvlcm": customer-a-bucket
    "m3vjrpgnvlcm5x8olkguh1l2": customer-b-bucket
    "vjrpgnvlcm5x8olkguh1l2m3": customer-c-bucket
  buckets:
    customer-a-bucket:
      bucket_url: "https://customer-a-turbopuffer.s3.us-east-1.amazonaws.com"
      provider:
        aws-irsa:
          region: us-east-1
      aws_assume_role:
        arn: "arn:aws:iam::123456789012:role/turbopuffer-access"
    customer-b-bucket:
      bucket_url: "https://customer-b-turbopuffer.storage.googleapis.com"
      provider: gcp-workload-identity
    customer-c-bucket:
      bucket_url: "https://customerc.blob.core.windows.net/turbopuffer-data"
      provider: azure-workload-identity
      azure_tenant_id: "00000000-0000-0000-0000-000000000000"
      azure_client_id: "00000000-0000-0000-0000-000000000000"

Creating additional clusters

You can create additional clusters by reusing your existing BYOC kit and control plane API key. Follow the deployment guide again with a unique cluster_name in values.yaml, and generate new turbopuffer org IDs and API keys for the new cluster with scripts/generate-secrets.py.

Applying configuration changes

After updating the configuration, apply the changes using the Helm upgrade command.