Jacob [0:18]:
My name is Jacob. I'm an engineer at Legora.
Simon [0:26]:
And I got a step into frame here. I'm Simon. I'm the CEO and co-founder of
turbopuffer, a search engine that we work with Legora and others on.
Jacob [0:36]:
Super quickly, an introduction to Legora. We're a collaborative AI platform for
legal work, which means we have law firms that are clients and we have in-house
legal teams that are clients. They use Legora to do reviews of contracts, to go
through an absurd amount of contracts and make sure that they all look good.
They use it to create new contracts and do legal research, which means looking
over all the potential law, and they collaborate inside Legora. So you can think
of Legora sort of as a Linear, Figma, Notion, and GitHub for legal work. It's a
lot. We are one of the fastest growing companies right now. We grow extremely
fast. Yeah, tons of numbers on the screen. I'll just skip through that. What we
really want to talk about is search today.
So at Legora, there are two types of search that we do: project search and legal research. Project search is basically projects in Legora, which is like the unit of work that you have. So if, let's say, you are SpaceX and you want to acquire Cursor, then that would be one project with your law firm. They would go into Legora and upload all these documents, and the law firm that helped you would go through all of the employment agreements, all of the contracts with suppliers. I know Cursor was using a turbopuffer; maybe there's a contract there they'd look at. Basically, you'd do the search confined to a project, and projects can be tens of documents to millions of documents.
The other use case is legal research. Legal research is sort of a deep research style workload where we'll search across tons of laws, previous cases, regulations, etc. People use this to answer questions such as, "How do we handle this specific thing?" They’ll also use it for litigation. Maybe they want to sue someone, or maybe they are getting sued, and they'll use legal research to support and help their case.
So if we start at number one, project search, we've been through a little bit of a ride here on how we do search, starting at, you know, hundreds of thousands of documents all the way into two billion documents. We've tried a lot of different things. First, we started with a very, very simple one, which is just a single Elasticsearch cluster for all of our search workloads. That worked relatively well. It was sort of a simple setup. All of the tenants, you know, our clients, our users would be on one big blob storage where we stored the raw documents and on one big Elasticsearch where we would do all of their searching, the indexing, and the searching. Super simple. Worked relatively well initially.
Then we wanted to enter the land of the free, and we got some new requirements. You know, Americans only want processing to happen within the U.S., and Europeans only want it to happen within the EU, and Australians only want it to happen within Australia. So we had to basically—well, here's a scaling graph—we had to move to multiple Elasticsearch clusters. What we actually did was take the entire setup and just iterated over the set that is EU, US, and Asia Pacific, so we just had this multiplied by three to four. Kind of annoying, lots of overhead, but it got us to where we needed to be.
Then the next iteration of the story is enterprise. So really big banks, the biggest law firms in the world, they have really annoying requirements. The number one they have is they'll ask for full physical isolation of all of their data. There's probably a little bit of a war on what physical isolation actually means, but it essentially means they want their own database. They also want customer-managed encryption keys, and what that means is they basically have a key vault where they have an encryption key and they give us access to read the key. We then use that key to encrypt and decrypt all of their data. What that gives them is they can just revoke our access to their key, and then we can't decrypt their data anymore, and so it's safe. In a way, that gives enterprises a lot of control over all of their data because they control the keys to reading it.
So we moved from Elasticsearch to Postgres, and I imagine a bunch of you guys are like, "Why would you ever put your vectors into Postgres?" It actually works surprisingly well, and the reason that we did this was we were already using Postgres for OLTP workloads. We sort of already had to do this split of multiple Postgres and multiple lots. It was really easy for us to try to shift all of our search into Postgres as well because then we only have one system.
So the setup here was PgVector, specifically disk ANN TS vector for the search, not BM25, which was, you know, we lost a little bit of retrieval performance there. What we do is partition the table where we store all the document chunks. We partition it super aggressively, like 4,000 partitions, and then for each project, we'd basically hash the project key and bin pack them into the partitions. That actually worked relatively well, but it was expensive, and search performance wasn't super, super good. What happened was when we scaled a lot, everything just broke and exploded.
You can imagine, like, you have a bunch of projects, and some of them you spin up a project, you work on it, and then you close it, and you basically never go back to it again. We have a bunch of those where they never get queried, and we have a bunch that get queried all the time because they're super active projects. When we packed them into partitions, the cold ones and the hot ones would land on the same ones, and the partitions would get really, really big. When we queried them, Postgres would pull the partition, put it into memory, we put stuff, and then we create another partition and another partition, and it essentially thrashed the cache all the time. What that meant was our latencies would spike. We went from search and ingestion P99 of 100 milliseconds into 20 seconds, which you can imagine is a really bad user experience.
So then we went to turbopuffer when we were about, I think, 400 million documents, something like that. What we did with turbopuffer was we did one namespace per project. The advantages of moving to turbopuffer are that we got BM25, real BM25, much better relevancy, much better latencies, and it was extremely simple to operate because we could just have a single turbopuffer cluster. We didn’t have to have a bunch of different ones like with Postgres, and since it's blob-based, it could just query the blocks that we had anyway. Much lower cost, and it was extremely simple to operate, and we didn't have this problem with the partitions because if a project's not used, it's just a block. It's really easy. And Simon can talk a bit more about why that works so well.
Simon [7:28]:
Yeah, so Legora has some of—and legal in general has—by the way, if Jacob and I
have similar accents and maybe even look a bit similar, it's because we're both
Danish. turbopuffer has a particular architecture that supports these kinds of
very regulated environments really, really well. But in order to understand
that, we have to understand what kind of search engine turbopuffer is and why it
is different from the ones that they used in the past.
Since the very beginning of turbopuffer, the design has more or less been the same. There may be changes in the future, but the design has stood the test of time. When you do a write to turbopuffer, we write directly to object storage. There is no disk replication, there's no Paxos, there's none of that—direct to S3. That's the fundamental trade-off in turbopuffer, right? Hundreds of milliseconds. If you're like Shopify and doing inventory reservations for a Kylie Jenner flash sale, not going to work. Very, very good for search. Because generally when you're doing search, doing a slow write is fine as long as the read performance is adaptable and good.
So that's what happens on write. It just goes into a write-ahead log. You can imagine you write 1.json, 2.json, 3.json. Obviously, it's a database, so it's not JSON, but for illustrative purposes, that's what happens. In the background, we build the vector indexes, the text indexes, the columnar indexes, and so on to satisfy the queries that Jacob and other customers have.
Then at query time, we can go in, and the query reaches some namespace. A namespace is kind of our concept of a table. You can think of it as a directory on S3 that's isolated from everything else. We go to the node that is most likely to have it. It could go to any node, right? It could go to every single node, and they're all read replicas, but it would go with some affinity to the node that has the highest probability of having it in cache. We check the memory cache for any objects, NVMe SSD cache, and then finally to object storage. Everything in turbopuffer is optimized around doing as much work in as few round trips as possible, right? S3 has a P99 on a one megabyte blob size of around 200 milliseconds; you want to do as few round trips as possible, right? Ideally, you do around three. Everything in turbopuffer, the database, is designed around minimizing the number of round trips.
This is also amazing for modern disks. If you do a lot of concurrency and few round trips, you utilize them optimally, and everything in turbopuffer is designed around this. So why is this so good for a company like Legora? Well, object storage native, if you design it around the atomic unit of separation being the namespace or the table, every single table could be encrypted with a different key. Every single namespace could be in a different bucket. We have customers that have thousands of buckets that they have namespaces in so that their customers get the warm IT fuzzies of having the bucket in their own cloud account. They can also be encrypted with their own keys. You can share buckets, you can do whatever configuration you need at the namespace level. You can re-encrypt with different keys, you can move them around, and you can re-encrypt with other keys.
For Legora in particular, this was really important for this full physical separation, right, an encryption separation. All of the namespaces needed to be physically at rest with different keys and as separate as possible. S3, GCS, Azure BLOBs, they pass that, and the other parts of the hierarchy also, except the NVMe SSD cache, because in the SSD cache, we consider that to be volatile like memory. But your customers did not, so what we did was we thought we were going to implement encryption into the disk cache, but instead, we just disabled the disk cache and saw how it fared. The performance of turbopuffer, even without the disk cache, with just the memory cache, was so good that we just kept it that way for some of the Legora workloads where we couldn't have the disk cache for multi-tenancy. turbopuffer will support that in the future, but it just goes to show the natural point where turbopuffer allows this encryption and storage and separation to become fully multi-tenancy native.
I'll hand it back to you on what happened then.
Jacob [11:38]:
And then, drum roll please, latencies look like this. Um, is my mic working? No?
Could I—or else that's screaming really loudly. It speaks for itself if you
can't hear me. Okay. Latencies improved an order of magnitude basically. And
these are median latencies, so P99 were even better. So obviously this is a huge
thing when you're doing—I mean, one thing is if you're doing a single sort of
RAG style thing, but if you have an agent that does twenty queries, a hundred
queries, these really, really add up.
So that was on the project side, and then a more recent thing is legal research. Legal research is a kind of a difficult problem, and the reason it's difficult is that the corpus is extremely big. So we're racing towards 10 billion vectors, and we're growing extremely fast. We also have quite high read, so QPS can spike a lot because we do a lot of fan-out. Like if you do a sort of legal research query, we will fan it out to a bunch of different queries and we'll keep going. The reason we do that is we need this heavy filtering because essentially it's kind of like a graph for a few different reasons. Firstly, it's hierarchical; you know, you have cities, counties, states, and federal law, and it's the same all around the world, so you need to respect that authoritative sort of hierarchy. There's also some temporal validity, so one judge might overrule a decision that's been made somewhere else, and you need to also respect that and figure that out. Sometimes there's even a new regulation that has exemptions or special cases of an old regulation.
And so if you're finding this one, you need to find all the other ones as well. So you can imagine that it sort of explodes the search. We started on Elasticsearch for this, but also moving to turbopuffer. Elasticsearch just got extremely expensive because we have to have everything there. But with turbopuffer, we can basically take different jurisdictions and make them namespaces in turbopuffer. That means some of them—here's an example—like you have the EU that gets queried all the time, that's super hot, and some of them, let's say Danish law, because we're Danish, no one cares really; it's such a small country, so it doesn't really get queried. That can just stay on blob, and that's fine. Because it's sort of a deep research style workload, if there's 500 milliseconds latency to fetch that cold blob, that's okay. That's fine. It's not really a big problem. So the way that turbopuffer is designed lends itself super well to this super long scale of cold weird namespaces and a few that are really, really hot.
Yeah, and Simon wants to talk more about that.
Simon [14:22]:
I was talking about why the company is called turbopuffer, another talk here
earlier today. But one of the other explanations of the name turbopuffer is that
it's about puffing into the different memory hierarchies and really mastering
when data should be in particular memory hierarchies. You can think about it
here, right? Something like EU law might be more or less part of almost every
one of the legal research queries, right? So that probably sits closer to NVMe
SSDs than memory, right? The economics kind of change as you move up and down
this hierarchy. In memory, you want things that are queried a lot, right? Then
the economics of memory are great. NVMe SSDs, you can do a lot of things
directly on them, but the economics change as you move up and down this
boundary. The latency changes, and the weight of the database is architected to
take advantage of it in terms of round trips versus random versus sequential—all
changes as you navigate this hierarchy.
turbopuffer is a database that is really designed around the memory hierarchy, and all of the smarts in turbopuffer is that all of these namespaces are puffed in and out of the cache. You can think of this as we want to spend as much time, have as much data pushed as far down in this hierarchy as possible to get the best performance-cost ratios. So how does that apply to search? Well, for something like a vector search, for example, there are two fundamental ways to do vector search. One is to navigate, basically design a graph. The problem with a graph on something like object storage or disk—again, we want to have things as far down that memory hierarchy as possible. The problem with a graph—this is not a graph; this is a tree—but in a graph, you have to navigate from the center of the graph. Every time you navigate through these nodes, you're doing 200 millisecond P99 to S3, right? You're trying to shrink the diameter of the graph; you're trying to do all these tricks to make the graph, but fundamentally you're at odds with the fact that a graph is about a random-sequential trade-off that you have in memory and in registers, but not further down the memory hierarchy.
The way turbopuffer does it is organize it into clusters. Vectors, you can think of in two dimensions just as points in a massive coordinate system, and we can organize them into clusters. turbopuffer then creates clusters of clusters and clusters of clusters of clusters to essentially organize all of the vector data in a tree. You can basically think of turbopuffer as a very, very complicated B-tree, right? Because it's a tree on this geometry of this entire space and the clustering of it in an approximate way. Now the root centroids further up the tree, you can imagine, are part of every single time you search, right? We're always trying to figure out which clusters that we're in, and we're always looking at the upper levels of the tree. So they're going to be further up the memory hierarchy, right? Closer to the registers, almost all in DRAM. Now the leaves that have all of the actual legal cases or whatever long document could be—could be images—all of that is probably going to be on SSDs with that single one millisecond round trip at the end. It doesn't make sense to have all that puffed into DRAM. This is fundamentally the cheapest way that you can run a database, period.
So for something like Legora or even web search, which is into hundreds of billions or tens of billions, depending on how much of the web you've scraped, this is fundamentally the cheapest way to do it. We have customers that are indexing massive parts of the entire web into turbopuffer, which is really also a part of what legal research is. Full text is also really respectful of the memory hierarchies. The way that text search works is essentially you can think of it as a hash map. You have a big document, and then you take every single one of the tokens and you put them into the key in the hash map. The value in the hash map is some set with all of the document IDs that have that term. So then if you search for "New York population," you're finding those three places in the hash map, and then you've taken the three sets and doing an intersect on the sets. While you're intersecting, you're also trying to do some kind of scoring, right? A document that has "York" in it is probably more valuable than a document that has "New" in it because "York" is a more rare word. When people say BM25, this is the scoring that they're referring to. The art of full-text search is one—we want to minimize the number of round trips. So first, you download the parts of the dictionary that are relevant—round trip one—maybe a round trip one before that to index into where the parts of the terms are, and then the second round trip is to get these massive lists. Try to make the list as small as possible by compressing them. But also while you're doing the text search, you're trying to minimize the amount—again, you can probably imagine that at some point there's a point where you've seen so many documents with "population" and "York" that have much higher scores that documents that just have "New" in them are irrelevant anymore. This is like a mega crash course in how text search works, and counterintuitively to most people, text search at web scale is more difficult and more computationally expensive than doing vector search.
I'll hand it over to you.
Jacob [19:16]:
Cool. So key learnings from what you heard today: retrieval is extremely
important to Legora. It's key to legal reasoning. turbopuffer really excels for
us because it makes it extremely easy to operate. We have 70 plus tenants. We
have 100, we have 200 tenants. You know, if we had to have separate
Elasticsearch databases for each of these, it would be just hell. But we can do
this natively with turbopuffer with data residency in CMAC, etc. And then it's
extremely cost-efficient generally when you have these types of workflows or
workloads that we do, where there's a long tail of cold indices basically that
you don't need to query so much, and you're sort of okay paying the small
latency cost for it.
So now with turbopuffer, and four seconds to go, we can focus on making Legora. We can focus on the product, making it really, really great, and not on scalability and infrastructure. Also, David, our CFO, is really happy about the cost. So it's great. Thanks, everyone.
