Building a Search Engine for Programmers

As a security analyst, when I'm trying to figure out what a malicious script / executable does, it often involves searching for weird strings I find in files that seem like they would be pretty unique. Or even specific sets of strings, even if it isn't very easy.

Google used to be awesome for this. Now it is still the best that I'm aware of, but it has gotten gradually worse over the years. So basically the best tool for a job is one I would describe as infuriatingly bad.

I think the problem is that it does too much to try to protect a person from malicious stuff on the web. It also does too much to guess what you might actually want instead of giving you what you asked for.

Probably the biggest single thing that Google has done to screw it up is that it no longer respects quotes. Maybe there's a workaround but I haven't figured it out. 10-15 years ago if I wrapped something in quotes Google would give me exactly what I wanted. Now its very finicky and 70% of the time it gives me what it thinks I want. These guesses are almost always wrong.

That being said, there is also a usefulness to being able to search for a code snippet or another string and get things that are very similar, even when they aren't an exact match. I think having multiple modes would be useful.

I think there might be some overlap between what I described above being useful for security researchers and what might be useful for programmers.

This is based on something Hillel Wayne wrote (@hillelogram) wrote on twitter not that long ago. The gist of it was it's ok that we're all using Google as part of our programming workflows, but why on Earth should we ever need to ask the same question twice? If you can find those threads, there might be more there than what I just said.

Obviously privacy is a concern here, but while I'm leery of Google knowing everything I do, I'm a lot happier with a technical search engine knowing which pieces of syntax I can't remember.

Also, the market definitely exists. A lot of time, if I am not sure how to formulate my question yet, google really sucks. It also regularly fails to find projects/tricks that I know exists and am able to find through my GitHub stars or browser history after some tedious browsing.

Often times I find an example or documentation, and I copy it to a playground to tweak it / experiment with whatever feature I am implementing.

Exception messages could be an important thing to focus on. That is the second thing i when search engines matter to me often (support fuzzy search here: abstract away the too concrete stuff but keep the actual message).

It would be great if you would understand versioning of documenations: it always takes me a while to understand if the docs apply to the version i am actually using.

So a combination of easy search of the wayback machine or a search in all online books.

For example if I search something related to numpy mmap in Python, in some cases only core numpy mmap answers make sense as results. In other situations, the question is about mmap generally and info from other languages or the core mmap definition could answer my question.

I don’t want to be checking boxes or toggles, or retrying the query with different text over & over to get the engine to understand these differences or when one class of answers is appropriate vs the other.

- Would be even cooler to detect the IDE, or even the error message itself (of course be careful not to leak sensitive information)

[0] https://news.ycombinator.com/item?id=23263918

[1] https://lobste.rs/s/dji0it/experimental_search_engine_for