We should pick experts that other experts believe in. If we happen to be experts in a field, or even adjacent fields, then by all means, let's pick experts based on our own understanding of their arguments. Knowledge does transfer to some extent, so if you're a mathematician or physicist or statistician, you could probably judge some climate or medical papers based on their statistical methods - especially when you find flaws (even if the math is perfect, if you don't know why they chose a particular model and how well it might actually fit the thing being modelled, you may not be sure they are not choosing a known bad model).

But if we're not, the only thing we can meaningfully do is to look at what other experts are saying is the right opinion, and judge based on popularity in their field.