The model still needs to attend to the prompt when generating the answer. Modern attention techniques help here, but for lots of simple queries most of the compute still goes into taking the system prompt into account, I guess.