I've heard it claimed that the era of being able to do this (buy slightly old used server hardware cheap on ebay) is coming to an end because, in the quest for ever more efficiency, the latest server hardware is no longer compatible with off-the-shelf power supplies etc. (there was more but that's the part that I remember) and therefore won't have any value on the second hand market.

I hope it was wrong, but it seems at least plausible to me. I'm sure that probably fixes could be made for all these issues, but the reason the current paradigm works is that, other than the motherboard and CPU, everything else you need is standard, consumer grade equipment which is therefore cheap. If you need to start buying custom (new) power supplies etc. to go along, then the price may not make as much sense anymore.