Let’s say I entered a website to search for the name of a book, the website contains millions of books, and it also supports searching by the content of books. In postgresql naive approach would be like this: LIKE '%something%’ maybe for 10-100 books it is fine, but for millions of books it is extremely slow

That’s when full-text search engines help us

In the big picture, we can divide searching into two stages:

  1. Index time - when documents are analyzed and added to the search index (usually on insert or update)

  2. Query time - this is where we search the data

Index time

1. Analyze text

Let’s say we have such text: “The best programmer in the world is Bakhrom.”

We should go through stages to make it better to search

A. Tokenization

Split text into tokens:

English-like: split on whitespace/punctuation

Some languages need special tokenizers (Chinese/Japanese), or compound word handling (German)

Result:

[ “The”, “best”, “programmer”, “in”, “the”, “world”, “is” “Bakhrom” ]

B. Normalization

Make matching consistent: Lowercase: Quick → quick Remove accents (optional): café → cafe Unicode normalization (important for weird equivalences)

Result:

[ “the”, “best”, “programmer”, “in”, “the”, “world”, “is” “bakhrom” ]

C. Stopwords