Export documents

To export all documents in a namespace, use the query API to page through documents by advancing a filter on the id attribute.

Documents inserted while the export is in progress will be included.

A common use-case for this is to copy your all documents to a different namespace after some client-side transformation. To copy documents without transformation, use copy_from_namespace for a more efficient server-side copy (follow with delete_by_filter to copy only a subset of documents).

import turbopuffer

tpuf = turbopuffer.Turbopuffer(
    region='gcp-us-central1', # choose best region: https://turbopuffer.com/docs/regions
)

ns = tpuf.namespace(f'export-example-py')

last_id = None
while True:
    result = ns.query(
        rank_by=('id', 'asc'),
        limit=10_000,
        filters=('id', 'Gt', last_id) if last_id is not None else turbopuffer.omit,
    )

    # Do something with the page of results.
    print(result)

    if len(result.rows) < 10_000:
        break
    last_id = result.rows[-1].id