That unencrypted bootstrap process can be modified by anyone with access to the disk, physical or remote. Theoretically, someone can inject a keylogger into the process and exfiltrate your encryption key's password, or a process that waits until you're decrypted and exfiltrates your data. It's also a potential vector for ransomware, root/boot kits, etc.