Mostly cost. The board itself, but also server boards are often hard to find for desktop sockets, so that drives cost for the CPU as well. But, if I get a server board, maybe I'm going to get one with IPMI, and those usually have serial consoles and some limited VGA for KVM over IP too.
I could afford a $800 server project, but I'd rather spend $300.