It's possible that the difference is in the actual code: x86 machine code sometimes has multiple different encodings for the same assembler instruction. These would show up as single-byte differences in the binary. See http://www.strchr.com/machine_code_redundancy for some examples. Compilers might use this to stenographically hide additional data in the binary. Printer manufacturers already do something similar (https://en.wikipedia.org/wiki/Printer_steganography). I wouldn't be surprised if MS compilers embedded a hidden data in binaries -- it could be useful to track down malware authors; or identify software created with pirated MS tooling.

Yes that is a good step towards fully deterministic build. There are ways to manipulate the PE file though to achieve slightly different behaviour. If you want to be thorough and make sure that your binary was not infected by malware for example then comparing the binaries makes sense.