Speaking as a case in point. PE lessons were my nightmare. My fitness was always below class average, and never had an interest in competitive sports - I always found exercise tiring, boring, and something you are forced to do. School cemented that attitude.

The lessons could have been better. For instance, starting with making one run as fast and as far as you can (and repeating that every few months) meant only that I wanted to puke my guts out. If we were following a running program of increasing difficulty instead, I might have noticed my stamina increasing and maybe would have liked it. Though our teachers probably assumed that we're doing sports after school anyway, and not e.g. sitting at home learning how to write video games.

I'm older now and I see it's about feedback. If something makes you feel miserable, then without strong sense of purpose about it (encouraging it is something school universally sucks at) you're likely to avoid it and hate it even more. OTOH, if you started your PE lessons relatively fit, they were probably pleasant for you and thus the feedback loop worked in your favour.