Reranking

Native reranking uses an ML model to reorder the results returned by vector search, full-text search, hybrid search, or another ranking function, so that the documents most relevant to the query come first. You don't need to integrate with a reranking provider or manage a separate API key. turbopuffer sends the candidates to the model and returns the reordered results.

An introduction to rerankers

Most search systems work in two stages. The first stage searches the whole namespace and narrows millions of documents down to a small set of candidates that are likely to contain the answer. If the answer is a needle, the first stage picks out the few haystacks it is most likely to be in. The second stage searches through those haystacks: a reranker reads each candidate closely and puts the most relevant ones first. We recommend adding this second stage to nearly every search application, unless it can't afford the latency of a model inference step on each query.

Take a simple example where the query is "How do I reset my password?". First-stage retrieval returns five documents, but ranks password policy and billing ahead of reset password and forgot password. Running the query and those five documents through a reranking model with rerank_by puts the most relevant documents first.

                ╔═ rank_by ══════════════╗   ╔═ rerank_by ════════════╗
                ║  top 5                 ║░  ║  by relevance          ║░
                ║ ┌────────────────────┐ ║░  ║ ┌────────────────────┐ ║░
┌────────────┐  ║ │ 1 password policy  │ ║░  ║ │ 1 reset password   │ ║░
│ "How do I  │  ║ │ 2 billing          │ ║░  ║ │ 2 forgot password  │ ║░
│ reset my   │─▶║ │ 3 account settings │ ║░─▶║ │ 3 password policy  │ ║░
│ password?" │  ║ │ 4 reset password   │ ║░  ║ │ 4 account settings │ ║░
└────────────┘  ║ │ 5 forgot password  │ ║░  ║ │ 5 billing          │ ║░
                ║ └────────────────────┘ ║░  ║ └────────────────────┘ ║░
                ║                        ║░  ║                        ║░
                ╚════════════════════════╝░  ╚════════════════════════╝░
                 ░░░░░░░░░░░░░░░░░░░░░░░░░░   ░░░░░░░░░░░░░░░░░░░░░░░░░░

        ┌─────────────┐
        │ "How do I   │
        │ reset my    │
        │ password?"  │
        └─────────────┘
               │
               ▼
╔═ rank_by ════════════════════╗
║  top 5                       ║░
║  ┌────────────────────────┐  ║░
║  │ 1  password policy     │  ║░
║  │ 2  billing             │  ║░
║  │ 3  account settings    │  ║░
║  │ 4  reset password      │  ║░
║  │ 5  forgot password     │  ║░
║  └────────────────────────┘  ║░
║                              ║░
╚══════════════════════════════╝░
 ░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░
           relevance
               ▼
╔═ rerank_by ══════════════════╗
║  by relevance                ║░
║  ┌────────────────────────┐  ║░
║  │ 1  reset password      │  ║░
║  │ 2  forgot password     │  ║░
║  │ 3  password policy     │  ║░
║  │ 4  account settings    │  ║░
║  │ 5  billing             │  ║░
║  └────────────────────────┘  ║░
║                              ║░
╚══════════════════════════════╝░
 ░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░

Reranking models judge relevance more accurately because they read the query and each document together as a pair. Embedding models, by contrast, encode the query and the documents separately, and BM25 scores documents by matching terms. That accuracy costs compute, and each stage spends more compute per document than the one before it. First-stage retrieval is cheap enough per document to cover the whole namespace. A reranker runs a model on every document it scores, which is affordable only because it sees just the candidates.

Here is the password example as an API request, retrieving 100 candidates instead of 5. First-stage retrieval is set by rank_by, and reranking by the rerank_by object, which specifies:

model: the reranking model to use
attribute: the stored text attribute to score
query: the text to score each document against

Reranking currently scores one attribute per query, and that attribute must also be listed in include_attributes.

from turbopuffer import Turbopuffer

ns = Turbopuffer(region="gcp-us-central1").namespace("docs")
query = "How do I reset my password?"

response = ns.query(
    rank_by=("text", "BM25", query),
    limit=100,
    include_attributes=["text"],
    rerank_by={
        "model": "voyage/rerank-3",
        "attribute": "text",
        "query": query,
    },
)

The response includes the document ID and requested attributes:

{
  "rows": [
    {
      "id": "password-reset",
      "text": "To reset your password, click 'Forgot password' on the sign-in page.",
      "$dist": 0.92
    },
    ...
  ]
}

$dist is the reranking model's relevance score, and it replaces the first-stage retrieval score. Results are sorted by this score, highest first. To return the first-stage score as well, see Keep the first-stage retrieval score. Scores aren't comparable across models, but with the same model and query they are comparable across namespaces and requests, so you can merge results and sort them by $dist.

Reranking improves the order only when rank_by has already retrieved relevant documents. The model only reorders that set, which means that a relevant document that is missing from it stays missing. A larger limit gives the model more documents to consider, but also adds more reranking work and increases latency.

Pricing

Reranking is billed per million input tokens processed by a model. The model scores each candidate as a query and document pair, so the query is counted once per candidate. How your query and documents translate into tokens is model dependent. A rough rule of thumb is that 4 bytes of English text is about 1 token.

See the list of available models for pricing. Standard query charges still apply.

Example: an application reranks every query with voyage/rerank-3.

Each query reranks 100 candidates of about 200 tokens each (roughly 800 bytes of English text). The query itself is about 10 tokens.
The query is counted once per candidate, so each query processes 100 × (200 + 10) = 21,000 tokens.
At 1 query per second, the application runs about 2.6 million queries a month (86,400 seconds a day × 30 days), or about 54 billion tokens.
At $0.05 per million tokens, reranking costs about $2,700 a month, on top of standard query charges.
Reranking 50 candidates instead of 100 halves that.

Models

5 of 5 models availableRegion

Model	Price $/1M tokens	Inference
`cohere/rerank-v3.5`	$0.10	US
`typesafe/jev-1.13.0` Preview	$0.042	US
`voyage/rerank-3`	$0.05	Global
`voyage/rerank-3-lite`	$0.02	Global
`zeroentropy/zerank-2`	$0.025	US

Context length is the maximum number of input tokens a model will consider when scoring a query-document pair, including any model-specific prompting.

Choosing the right model

A reranking model can perform well on public benchmarks and still be a poor fit for your search. Your terminology, document lengths, and users' expectations all affect relevance. A small evaluation on your own data helps you see whether reranking makes a useful difference.

Start with a handful of queries that reflect how people use your application. Identify which documents would help answer each query, including queries your search already handles well so you can catch regressions. You don't need to label your entire dataset: inspecting a few result sets is enough to start seeing where the ordering could improve.

Run each query without rerank_by as a baseline, then try different models while keeping first-stage retrieval unchanged. Keep the namespace contents, rank_by, limit, include_attributes, rerank_by.attribute, and rerank_by.query the same; change only rerank_by.model. The model is chosen per query, so switching models doesn't require a reindex. Compare the documents in the first five or ten results, rather than their $dist values: scores aren't comparable across models.

The table below shows common outcomes and what they tell you:

What happened?	Before reranking	After reranking	What it tells you
A relevant document was buried in the results	40th of 100 candidates	2nd	Reranking made it more likely that the user would find the answer.
Search already put the relevant document first	1st	1st	This query gained no improvement in the document's position, but still incurred reranking latency and cost.
A relevant document moved down	2nd	15th	Reranking made this query worse. Keep these regressions in your comparison.
The relevant document wasn't retrieved	Missing from the 100 candidates	Still missing	Improve first-stage retrieval, for example by increasing `limit` or trying hybrid search. Changing the reranker cannot recover a missing candidate.
No document in the namespace answers the query	No relevant result	Still no relevant result	As in the previous case, a better reranker can't fix this.

Once a model consistently improves the results, compare its added latency and cost to decide whether the improvement is worthwhile for your application.

Examples

Keep the first-stage retrieval score

Because $dist contains the reranking score, the score from rank_by isn't returned by default. To keep it, add the same expression as a computed attribute. Computed attributes don't change which documents are retrieved or how they are ranked.

from turbopuffer import Turbopuffer

ns = Turbopuffer(region="gcp-us-central1").namespace("docs")
query = "How do I reset my password?"

response = ns.query(
    rank_by=("text", "BM25", query),
    limit=100,
    include_attributes=["text"],
    compute_attributes={"bm25_score": ("text", "BM25", query)}, # same as in `rank_by`
    rerank_by={
        "model": "voyage/rerank-3",
        "attribute": "text",
        "query": query,
    },
)

Each row then includes both scores:

{
  "id": "password-reset",
  "text": "To reset your password, click 'Forgot password' on the sign-in page.",
  "bm25_score": 7.4,
  "$dist": 0.92
}

For vector search, use a VectorDist expression to return the vector distance instead. In a multi-query, a computed attribute with the same name must have the same definition in every subquery.

Search titles and rerank using document bodies

The query you rerank with doesn't have to be the query you retrieve with. Because a reranker reads the query and each document together, it can use detailed context to judge which document applies. If your application has that context on hand, pass it in rerank_by.query and keep a short query for first-stage retrieval.

In this example, a support agent searches a knowledge base for "refund" while a customer ticket is open. First-stage retrieval matches "refund" against article titles, and the reranker scores each article's body against the ticket and the agent's query. The example assumes title has full-text search enabled and body is a stored string attribute.

from turbopuffer import Turbopuffer

ns = Turbopuffer(region="gcp-us-central1").namespace("support-articles")
query = "refund"
ticket = "I was charged twice for my March invoice. The duplicate shows as pending on my card. Can you reverse it?"

response = ns.query(
    rank_by=("title", "BM25", query),
    limit=100,
    include_attributes=["title", "body"],
    rerank_by={
        "model": "voyage/rerank-3",
        "attribute": "body",
        "query": f"{ticket}\n\nQuery: {query}",
    },
)

Several articles have "refund" in the title, including "Refund policy" and "Reversing a duplicate charge". The titles alone don't say which one applies, but the ticket does, and it gives the reranker what it needs to rank the duplicate-charge article above the general policy.

The ticket is sent to the model once per candidate, so a long rerank query multiplies tokens by limit. It also counts toward the model's context length for every pair, which leaves less room for the document body. Reranking also cannot recover an article that the title search missed, such as one titled "Duplicate charges" that never says "refund". To use the ticket in first-stage retrieval as well, add an ANN subquery that embeds the ticket, using the multi-query pattern in the hybrid search example.

Hybrid search with native embeddings

Retrieve candidates with both BM25 and vector search, then rerank their combined results. This example assumes text has both native embeddings and full-text search configured.

from turbopuffer import Turbopuffer

ns = Turbopuffer(region="gcp-us-central1").namespace("docs")
query = "How do I reset my password?"

response = ns.multi_query(
    queries=[
        {
            "rank_by": ("text", "BM25", query),
            "limit": 100,
            "include_attributes": ["text"],
        },
        {
            "rank_by": ("text", "ANN", ("Embed", query)),
            "limit": 100,
            "include_attributes": ["text"],
        },
    ],
    rerank_by={
        "model": "voyage/rerank-3",
        "attribute": "text",
        "query": query,
    },
    limit=10,
)

results = response.results[0].rows

Each subquery retrieves up to 100 candidates. turbopuffer merges the results, deduplicates documents by ID, and scores each document once. The outer limit returns the best 10 after reranking. Set rerank_by on the outer request and include the text attribute in every subquery.

You don't need to apply RRF first: model reranking scores the combined candidates directly. RRF is another way to combine subquery results, using their ranks rather than a model.

Limitations

Reranking requires a stored string attribute. Include it in include_attributes in every query. You can return other attributes alongside it.
You can return computed attributes alongside reranked results, but you can't rerank on a computed attribute. Aggregation queries are not supported with model reranking.
Documents with a missing or null reranking attribute are not sent to the model and receive a score of zero.
Reranking adds a model inference step after first-stage retrieval. Latency depends on the model, the number of candidates, and the length of their text.
Model availability varies by region. Use the region selector in the model table to check your region.

If your use case is blocked by one of these limitations or you'd like to use a model that isn't listed here, contact us.

Rate Limits

The default limit is 2,500 candidate documents per second per organization and model, with a burst capacity of 10,000 documents. These limits are separate from embedding limits. Reranking 100 candidates and returning 10 counts as 100 documents; duplicates across subqueries count once.

Rate-limited requests fail in full with HTTP 429 Too Many Requests. Contact us for higher limits.

Security

We work with external partners to provide access to reranking models. When you use reranking, the query in rerank_by and the selected text attribute from each candidate document are sent from our cloud account to the model's provider. Other returned attributes are not sent. The provider depends on the model you select. See our subprocessors page for our inference providers.