Or the Verilog/VHDL for the logic gates used by the CPU, for that matter?
Ideally, yes. But you still will be more secure than having multiple security holes.