Building a search engine

So I built a search engine. In two and a half weeks. During Christmas.

Yes, I have a life. Thanks for asking!

Following a course on Brilliant, I decided to code my own full text search engine. How hard can it be? 3K lines of code in (for just the backend), I can confirm, it’s educational, but a pain in the ass.

The basics

The thing in common between search engines is their index. That’s how they quickly know where to look for the searched content.

They achieve this through storing map of words and the documents they are in (and sometimes also exactly where each occurrence is). Now, we can quickly (by indexing the map) query where the word we’re looking for is. Then, read the data and return the hit with context.

I’ve expanded these principles a bit to provide a nicer searching experience, what you’ve grown used to in DuckDuckGo and Google.

Goals

As any good project manager, I set up some goals before starting the theory work and coding.

• Prioritize memory/disk use over performance. This implies searching through each of the matching files, for every query.
• Typo tolerance. You should be able to get results if you search for the start of a word or if you missed some characters.
• Fast enough for updating the results for each key press. The server shouldn’t be the b