Nikhil Benesch [0:12]:
All right. Hello, everyone. Thank you all for coming. I'm Nikhil, CTO of
turbopuffer.
Aaron Linsky [0:18]:
And I'm Aaron Linsky, the Head of Product Engineering at Bridgewater Associates.
Nikhil Benesch [0:22]:
And we're here to talk to you today about how you build an AI analyst that can
keep a secret. I'll explain a bit more about what that means. Now, this is a
very oversimplified view of how you would go about building an AI analyst, but
it is meant to capture an important dimension around permission. In this
diagram, your user interacting with this AI agent starts with a question, maybe
something like, "Why have token prices come down so much in the last month?" You
feed that question into an LLM, which will begin thinking about that question,
investigating, researching. As it does that investigation, it's going to search
for facts to ground its information, to lead its investigation. Some of those
facts might come from the public web, but other facts might come from private
documents in your organization. The public web is, on one dimension, much easier
to search than the information that's private to your organization. By
definition, the public web is public; everyone in your organization can see
exactly the same documents. So there's no permission issue there. Every public
web document can be immediately fed into the AI analyst no matter who's asking
the question. But the information internal to your organization looks very
different. Every email, every meeting note, every report might have a completely
different set of permissions attached to it, belonging to different groups,
maybe individual permission assignments on those documents. You can see here
we've got different colors reflecting the fact that every document in that
private data set has a different set of permissions. This is a classically hard
problem in information retrieval, and it's something we explicitly designed
turbopuffer to solve. But before I get into that, I'm going to hand it over to
Aaron to talk about why this private information is so important to
Bridgewater's AI analyst.
Aaron Linsky [2:07]:
Great, thank you. So first, I'm going to talk a little bit about who Bridgewater
is, what we do, and then we'll get into how we're using turbopuffer and the
needs there. For the past fifty years, we've pursued the best understanding of
how markets and economies work. We then encode that systematically with logic
into a system that we can stress test over time. What you see here on the left
with the yellow legal pad is our bond system from 1980. It's a set of if-then
statements that we used to trade bonds back then. Over the next few decades,
we've evolved that process from a yellow legal pad to Excel to eventually Scala
and now Python. This research circle, as we call it, has five steps: Perception,
which is seeing what's going on in the outside world; Hypothesis, which is
coming up with ideas of why that might be true; Investigation, which is the
research of seeing if that thing is true; if it is true, we then systemize and
integrate it with our system and industrialize it to run every day and trade on
that. That's been our learning process. I've been at Bridgewater for about 20
years, and that's been what we've been doing with humans at the center. We're
now moving towards putting machines at the center and running that process. In
order for both humans and machines to work well, we need access to high-quality
data. We need both time series data, which we've spent decades creating a great
bi-temporal macro time series database that we call DeLorean, and more recently,
we've been creating an unstructured database that we call the Vault. So what is
the Vault? Today, it's about 100 million documents that we index, organize, and
secure, which we think represent some of the best third-party external research
and also our internal research. We've sourced from 135 different external
sources, which is a huge deal for both our legal and data teams to negotiate and
get them in. All of those sources have some sort of permissions around them,
where they might not be open to everyone at Bridgewater. We also have our
internal sources of unstructured data, which include our own writing, meeting
recordings, and internal reports. Both of those go into the Vault, and then both
humans, investors, and our AI agents read out of the Vault. What I haven't shown
here is that it circles back. The investors and the agents will create new
content that goes back into the Vault, so it's constantly improving. Here's a
very high-level view of what permissions in the Vault look like. I have a user,
we'll just say their ID is user one; they're in groups B and C. This is
incredibly simplified, unfortunately. There's a set of all of our documents
where it could be either permission to them via groups or directly. Here we have
the actual code of us going ahead and creating a query to try to look for some
data with whatever that query vector was that matches the viewers and groups.
This is simple in one way, and actually, when we first implemented this without
turbopuffer, we thought it worked fine. But then pretty quickly, as we started
to add more and more documents to the Vault, it fell over tremendously. What I
did then was look for other possible providers. We found turbopuffer, tested
them out, and they had a solution which Nick is going to go over right now.
Nikhil Benesch [6:25]:
Awesome. Thank you, Aaron. Let me very quickly talk about what turbopuffer is.
We are an extremely fast, extremely scalable, and extremely cheap search engine
built entirely on top of object storage. What that means is we write directly to
S3 or GCS or Azure Blob Storage, whatever your hyperscaler provides. That gives
us S3 economics. By default, your data is inert, cold on S3. Only when you start
querying it do we pull it into more expensive storage mediums like SSD or
memory. This is really powerful because once a query is hot, once the data a
query is accessing is in the cache, it's fast. If you're not accessing a
particular namespace or a particular region of a namespace, that data can just
hang out on object storage where you pay rock-bottom prices for it. This
architecture allows turbopuffer to scale cheaply to terabytes of data, which
would cost tens of millions of dollars in other systems, and we can often do it
at a tenth of the cost, with performance that's comparable, once you get the
data hot, to what you get with traditional disk-based search systems. We support
all types of search: keyword search, regex search, and what Aaron was showing
earlier, vector search. We're really not opinionated about how you search your
data in turbopuffer; we want to do search generically. But for the rest of this
talk, I'm going to focus on vector search because it's good to zoom in and go
deep on one particular type of search. Let's do a little bit of napkin math.
That's something we love to do at turbopuffer; it helps us anchor in the
simplest possible solution. We never want to introduce complexity to turbopuffer
that's not justified. One of the justifications for taking on complexity is if
you need it for performance. The absolute simplest way to do search is to put
everything in a list, and every time somebody shows up saying, "I want queries
that match this," you look at every single piece of data. Let's conservatively
say that every document you store in turbopuffer is 1 kB. This is incredibly
conservative. A vector embedding is often 1024 dimensions, so if you're using
just a byte per dimension, that's a kilobyte right there. In practice, documents
are often larger than this, but for the sake of example, let's say it's a
kilobyte per document. Even if you had all of that data resident in memory and
you were ripping through it as fast as you possibly could, which on modern
hardware is about 20 gigabytes a second, once you get up to 100 million docs,
that's going to take five seconds. For most search workloads, that's way too
slow. What we see in practice is most people want their search results to be
about 100 milliseconds. So the naive solution here does not work. If you've only
got 10,000 docs, great. That's going to take half a millisecond. turbopuffer can
do literally anything, and it will be fast enough. But as you get to scale, once
you're at 10 million or 100 million, you really need to start thinking about how
you represent your data so that you're not crunching through all of that. Of
course, I gave you the numbers for memory here; that's only if the cache is hot.
If that data is on S3, you're looking at a minute or more to pull that data off
S3, so this just isn't viable at the scale that Bridgewater operates.
So step one in turbopuffer is to build a vector index. This is something that we handle under the hood for you. When you store vectors in turbopuffer, we automatically build this index behind the scenes once you get enough data to justify it. Vector embeddings, if you're not familiar, are the internal numeric representation of concepts in LLM space. We like to use this example of red shoes and maroon sandals. You as a human know that these are very similar concepts and much more similar to one another than a concrete block is. Maybe you're thinking about shape, size, texture, and weight. The LLM can translate that into this numeric vector. Here I've only got two dimensions shown so that we can think two-dimensionally, but in practice, there are often thousands of dimensions. 1024 dimensions is a very common size. Thinking multi-dimensionally like that is very hard, so for the sake of example, we're just going to imagine that there are two dimensions here. Now, we don't actually know what those dimensions represent, but again, you can think of it as size and color. You can plot that on a coordinate system. So red shoes might be at the top left along with maroon sandals, and then concrete block is going to be down at the bottom right. I'm going to simplify wildly here, but a very workable vector index in two dimensions is just to plot things on a graph. You can zoom in; if you're looking for red shoes, that's the Q on this diagram. You plot that on this two-dimensional graph and zoom in to look for the vectors around it. Because of the way these vectors are created by the LLM, vectors that are near each other represent similar concepts and good search results to return. If we were just doing similarity search, great. We look in our two-dimensional coordinate grid, find some nearby vectors, return them, and life is good. This saves us a lot of time; we don't have to search the whole vector space. We can zoom right in to the documents that are going to be relevant to the query. So that napkin math from earlier, we can ignore. We're maybe only looking at 10,000 vectors here, and we know we can do that in a millisecond or two. The problem is when you add in these permissions. Not all of the documents that are most relevant for a given query are going to match those permissions. You can see in this diagram here that the minuses don't match the permissions, while the pluses do match the permissions. Our vector index is not immediately helpful to us, and we've invested a lot of time and energy at turbopuffer into solving this problem. Naively, you might try a post-filtering strategy. The way that works is the query comes in, you get maybe the 10,000 closest vectors, and then you throw away all the ones that don't match. This does not work very well because it doesn't provide the results that you're looking for. If I'm the user that only has access to that data in the bottom right and I search for Q, I still want to see the documents that were closest to Q from my perspective that include the results that I can see. But when you do post-filtering, you kind of spiral outwards, and then at some point you have to give up because it's been a couple of seconds and you haven't found any docs that match, or you haven't found the top hundred docs that match. Maybe you were only able to find 40. This makes users unhappy because they might know that a doc exists with maroon sandals, and they were searching for red shoes, and you don't turn it up. So you have very poor recall. Instead of starting with the vector index, what if we start with the filters? turbopuffer builds an index of all of the documents that match the filters. So Aaron can show up and say, "Hey, show me all of the documents that I have access to." Then turbopuffer can scan over all of those documents and see which of them are closest to the query vector. But what if Aaron has access to 50 million out of the 100 million documents in the Vault? Now we're back to square one. We're back in that napkin math problem where it's going to take multiple seconds, if not minutes, to get through all of that data. We're not actually using the vector index. If you look here, you can see we're just finding all the docs that match the filters and then scanning through them sequentially. So this also doesn't work very well. The trick to doing well on this problem is to do both. turbopuffer's vector index automatically clusters similar vectors. You can see there's a group of documents here that are similar, labeled five, and another group labeled zero, and another group labeled one. When we build these permissions indexes, we say, "Well, Aaron has access to documents in Cluster 5 and maybe Cluster 2, but he doesn't have access to any documents in Cluster 3 or any documents in Cluster 0." So we can just throw those clusters out; we don't have to consider them at all. First, we find all of the clusters that Aaron has access to, i.e., he has access to at least one document in those clusters. Then we search the closest clusters and scan sequentially inside those clusters. In turbopuffer right now, the clusters have about 200 documents. This is something we're always tuning. Over time, we'll probably increase that number to maximize throughput. But this basic strategy fundamentally makes search at scale with permissions possible. The trick is to combine pre- and post-filtering into what we call native filtering. The result is that across 450 million vectors, those 100 million documents turn into 400 million vectors, and turbopuffer is able to achieve 120-millisecond searches. That's the p90 latency across all of those namespaces by employing this sophisticated native filtering strategy. With turbopuffer, you show up with your permissions logic in your conceptual model, and you do what I think is a very simple translation of those permissions into turbopuffer filter language. Hopefully, you can directly map this code sample here: viewers contains one, groups contains any BC. You can see how you would project that under your organization's own permission structure. That turns into this very complicated systems engineering challenge under the hood, but turbopuffer hides that all away from you so that you don't have to think about it. You can just express the logical filters that represent your permissions in a fairly direct way, and we handle the complicated query planning under the hood to make that search come back in one to two hundred milliseconds, even at a hundred million document scale. Now, we're engineers at turbopuffer, so there are some limitations here. In particular, when you have nested groups, things get tricky. I'm not going to have time to go into that in too much detail, but if you have problems that look like that, please come talk to us. We have a lot of experience working through these, and we're always looking for ways to improve the product so that stuff is handled more automatically. Permission filters are really just the tip of the iceberg. In turbopuffer, we're trying to build an isolation-native search engine. That means whether you have fine-grained permissions within namespaces, whether you have security or regulatory compliance obligations that force you to regionally lock data, keep that data in your own VPC, or encrypt that data potentially not just with an encryption key you control but that your own customers control, turbopuffer has solutions for all of those levels of isolation. Permission filtering is just one piece of the puzzle in building isolation-aware search in turbopuffer. That's all we got. That's how turbopuffer and Bridgewater work together to do fast isolation-native search at a hundred million scale. Thank you, everyone.
