Rust Search
Find the best content on Rust, curated by the community; a
search engine for Rustaceans.
Quickwit is an opensource distributed search engine. The most challenging component of a search engine is its indexing pipeline. It takes a stream of JSON documents, processes them in batches, and emits index artefacts we call splits. We needed our implementation to be robust, testable, observable, and efficient... But even more importantly, our code had to be easy to work with. I will explain why we chose to develop our own actor framework and discuss the unique features of our implementation.
Follow me as I walk through my journey of building lnx, a distributed search engine written in Rust akin to Elasticsearch or Algolia that aims to be faster and more efficient using tantivy. I cover the challenges and solutions I encountered while developing lnx over the last year, the tradeoffs made and how you can build your own search engine using the tools the Rust ecosystem provides.
"My favorite new double-meaning programming phrase: "my c++ is a little rusty""