Later, when you actually reformat the drive, you (or your OS) should spend some extra time writing over the sector(s) containing your encrypted FDE master key. But only those sector(s) need to be securely wiped. Even if someone can brute-force the password you used, they won't have that sector to use to turn it into a decryption key for the sectors; and they definitely won't be able to brute-force the FDE master key itself.
And, to make matters better, in systems with a TPM (most, these days), the TPM holds the FDE master key, and does the job of "shredding" it when it's no longer needed itself, as an async job kicked off by the OS along with the reformat. Also, in such systems, the FDE master key is never loaded into RAM, because the TPM itself does the sector-wise encryption/decryption as a DMA process.