Took me really long to understand what this is... let me see if I get it right:
* You scrape rubyweekly.com into Sqlite entries.
* You provide a simple search engine on top of that.
1) Less is more, you can simply state that on your page ("Search an index of scraped rubyweekly articles).
2) If I get it right you are doing regex match for keywords, but maybe you could rely on FTS5 since you are using Sqlite, you may be able to get better results faster, maybe with less code? ([1] random article I found about rails+FTS5).
Cheers!
--
1: https://mariochavez.io/desarrollo/2023/09/01/full-text-searc...
--
The entire thing could be a static site using the above method -- even if it's many hundreds of megabytes in size!
``` api, openapi, automation, rest, graphql, rpc, soap, webhook, scrap, event-driven, serializ ```
What followed was a wealth of articles full of knowledge, understanding of his niche, and potential for connecting to potential leads and partners. Instead of calling it a day, he decided to share the tool with us, the people (which basically means his 100 minute hack turned into a 20 hour project if you include all the side quests and write-ups related to it).
https://slothfinder.apishop.io/
He has also open-sourced it on GitHub. Here are some reasons for you to look at the code:
- You want to see how to deploy a Rails project on the cheapest [render.com](http://render.com) non-free plan running on SQLite (making it cheaper than usual). After I already embarked on render.com, I found that [someone else documented how they experimented with something similar and ran Python on SQLite on fly.io for basically free](https://github.com/irskep/cheapo_website), so this might be another solution if you are interested in hosting your hobby projects. - You want a simple Ruby scraper template to scrape simply structured sites like xlcd.com/1, xkcd.com/2, etc. - You wanna see a logger that writes to STDOUT and logs (useful for local scraping if you want to watch progress but also write to logs for search purposes later). - You want to see a Turbo stream in action. - You want to see how a „first thing that came to mind search over a small ActiveRecord collection“ was implemented.
Ok writing first-person again. If you find the search useful, let me know if you would like something similar for the other publishings of cooperpress(https://cooperpress.com/publications/). They have top notch stuff on JavaScript, Postgres, and much more! I’ll also take suggestions for other scrapeable publishings that don’t have a search.
Enjoy
