Navigation
turbopuffer supports BM25 full-text search for string attributes. This guide shows how to configure and use full-text search with different options.
First, configure an attribute for full-text search in the schema when upserting documents:
Currently, the vector attribute is required for full-text search. We'll remove this requirement in the future. If you only need full-text search, set the vector to a random float.
{
"ids": [1, 2, 3],
"vectors": [[0.1, 0.1], [0.2, 0.2], [0.3, 0.3]],
"attributes": {
"content": [
"The quick brown fox jumps over the lazy dog",
"A quick red fox darted through the forest",
"No foxes were involved in writing this text"
]
},
"distance_metric": "cosine_distance",
"schema": {
"content": {
"type": "string",
"full_text_search": true // Enable BM25 with default settings
}
}
}
Perform the BM25 search using the rank_by parameter:
{
"rank_by": ["content", "BM25", "quick fox"],
"top_k": 10,
"include_attributes": ["content"]
}
This example shows how to:
[]string attributeFirst, upsert the documents with the schema FTS configuration for the fields:
{
"ids": [1, 2, 3],
"vectors": [[0.1, 0.1], [0.2, 0.2], [0.3, 0.3]],
"attributes": {
"title": [
"Getting Started with Python",
"Advanced TypeScript Tips",
"Python vs JavaScript"
],
"content": [
"Learn Python basics including variables, functions, and classes",
"Discover advanced TypeScript features and type system tricks",
"Compare Python and JavaScript for web development"
],
"tags": [
["python", "programming", "beginner"],
["typescript", "javascript", "advanced"],
["python", "javascript", "comparison"]
],
"language": [
"en",
"en",
"en"
],
"publish_date": [
1709251200,
1709337600,
1709424000
]
},
"distance_metric": "cosine_distance",
"schema": {
"title": {
"type": "string",
"full_text_search": {
"language": "english",
"stemming": true,
"remove_stopwords": true,
"case_sensitive": false
}
},
"content": {
"type": "string",
"full_text_search": {
"language": "english",
"stemming": true,
"remove_stopwords": true
}
},
"tags": {
"type": "[]string",
"full_text_search": {
"stemming": false,
"remove_stopwords": false,
"case_sensitive": true
}
}
}
}
Now you can search across multiple fields:
{
"rank_by": ["Sum", [
["title", "BM25", "python beginner"],
["content", "BM25", "python beginner"],
["tags", "BM25", "python beginner"]
]],
"filters": ["And", [
["publish_date", "Gte", 1709251200],
["language", "Eq", "en"]
]],
"top_k": 10,
"include_attributes": ["title", "content", "tags"]
}
For hybrid search combining both vector and BM25 results, see Hybrid Search.
For all available full-text search options, see the Schema documentation.
