The thing it, while many of the people who vote Tory want the UK to leave Europe, the people who actually fund the Conservative party make a lot of money from being part of the EU. The disconnect between the voters and the funders of the Conservative Party has been a defining feature of theirs for decades.
This is why mentioning Europe is so destructive for them, because they have to stay in Europe, but cannot publicly admit that.