OpenWRT/LEDE is a thing. But don't expect cheap home routers to be able to run useful services-- the cheapest hardware is truly bottom-of-the-barrel, even OpenWRT had to drop support for lots of devices.

Ubiquiti makes decent hardware, and now has an upgrade to go from their flavor of linux to debian stretch.

I wonder if Ubiquiti would consider selling a "developer" module of the Edgeroute-X or similar with more ram.

Or maybe look for a raspberry pi clone with dual ethernet (not connected by USB). Add a 128GB microsd for $20-$25 to have some room for video, photos, email, chat logs, etc.