HP (?) tried pushing the home server pretty hard several years ago, but I don’t think it ever took off, as I haven’t heard about it in any mainstream sense in years. But to your point, it was just running Windows Server, which isn’t exactly consumer friendly.
For something like a home server to take off it really needs to have that killer app. For most that would probably be something like Plex/Jellyfin, but your average user is just going to sign up for streaming services instead. And if a media server is all you want, Synology can take care of that pretty well. I recently moved my Plex server from a Mac mini (which I used in some form since Plex was first launched) over to a Synology NAS using Docker. Of course that’s not very user friendly, but assuming the native Plex app is better for the average Joe.
With so many cloud providers that make it easy to get to your data anywhere in the world via your laptop or phone, it’s hard to argue in favor of moving to home servers for much of anything. Sure, you can access it remotely if you set it up, but it’s going to be more hit and miss, and when things don’t work you’re just stuck hoping you can fix it.