My intuition would have been that you are better off using a fairly standard transceiver and spending your engineering budget either increasing power or getting a bigger dish (either by launching on a wider rocket or with a folding design).

Lasers might interesting for the downlink, but receiving a laser signal on the probe sounds difficult (earth is pretty bright).