And yes, when there's a task that one person happens to know most, people will often defer to them. But that in itself is educational, as the experienced dev explains why the given task is easy/hard. And every task is different, so the person you're deferring to will be different, and you still often get the two or three people who know the task best disagreeing until they hash it out, etc. And very often it's another person who can point out "well it would be that easy except three sprints ago I did x and so you'll now need to do y...". And of course plenty of tasks really are brand-new so everyone's figuring it out together.
If you're really not having actual discussions around complexity in planning poker, then the facilitator/lead/manager might be doing it wrong. You do have to create an environment where people are expected to speak up and disagree, to demonstrate that this is welcomed and expected and rewarded, and not just some kind of checkbox exercise where the most senior dev gives their estimation and everyone agrees. This is also a reason why it's literally done with cards where everyone is forced to put their number on the table at the same time, so that you don't wind up with some senior person always going first and then everyone else just nodding and agreeing.