Fully support your dream -- and out of similar reason/sentiments I created GateMate PC (google for it) and GateMate S/359. I will return to these systems in autumn; it's a pure joy to work with them.

As for the video controller -- if during the next couple of days I have zero success with GK-208 initialization in U-Boot, I have a plan B. I will throw away that card completely and will buy DragonBoard (Xilinx Artix 7). I already have a video controller implementation for it.

Own video-controller + own OS + own bootloader on RISC-V. What could be better! :D