Security researchers like Matthew Green seem to care[0], the Signal people surely do, I myself do, too. Isn't that enough to raise that question?
> if you're paranoid enough to worry about it
You make it seem like that's an outlandish thought, when in reality there have been tons of reported vulnerabilities for SGX. And now QC represents another risk.
> it's not like Signal would be less secure than any other third party if SGX were broken
That's a weird benchmark. Shouldn't Signal rather be measured by whether it lives up to the security promises it makes? Signal's whole value proposition is that it's more secure than "third parties".
[0]: https://blog.cryptographyengineering.com/category/signal/