For example I'm an expert on real time video streaming in embedded systems. It's trivial to me to program a real time system that will stream video at a lower resolution in real time from scratch. I pass that problem to your average backend web developer they're going to take 100x longer then me.
The issue is, while it may seem obvious to you that the domain is too specialized, when you yourself spend to much time in a specific domain you become biased and you start to think such knowledge is basic because you get too good at it. It's invisible to you.
Time shouldn't be the factor measured here because people have such varied backgrounds.
The point of the assignment isn't to make it "fair" (in some loose sense), but to filter out candidates.
Let's say I come up with an assignment and I pass it to a junior to see how long it takes. The junior does the work, likely reports the time rounded down, but also with no stakes. Jim job is secure.
As an interviewee you're gonna spend time polishing the code. Every variable well named, lots of comments, good functions, well considered parameter order etc. It's gonna take longer because , for you, the stakes are higher. You'll likely build some code to test it, and so on.
This is before we discuss domain knowledge that the junior had, and also that the company implemented the task in the first place and didn't just spitball the time expected.
The reason they end up getting it wrong is usually because they pick something domain related to what the company does. So these guys literally work in that domain every single day of course they know it in and out.
When they throw it at someone completely foreign to the domain it's going to take a longer time to figure it out. Most interviewers just lack the ability to see this.