Inference is often done fixed point, whereas training is (usually) floating point.

Inference also prefers different IO patterns, because you don't need to keep the activations for every layer ready for backpropogation.