I would argue that it is a very popular film released by a major studio, and directed by one of the most popular directors in history. Mulholland Drive is not a film that only "film buffs" saw. I'm not sure if I've every met anyone who hasn't seen a David Lynch film, and I rarely meet anyone who hasn't enjoyed a David Lynch film. He's a multi-millionaire in high demand that entirely dictates his own work conditions.
Count me in your second group. I've seen several David Lynch movies, never enjoyed a single one of them. There's something about his film aesthetic but also about his view of what the world really is that just doesn't sit right with me. When I do see them it's because one or more film nerds I respect has raved about it, and I think I should give it a chance. But ... nope, same film aesthetic, same worldview that I just can't enjoy.
Of the people I know who have seen a David Lynch movie, I haven’t met anyone who enjoyed them. Speaking for myself, Mulholland Drive is definitely in the running for worst movie I’ve ever seen.