Eccentric hypertrophy (athlete's heart) is the positive adaptation resulting from training the heart. The heart has a lower resting rate and is more efficient at pumping blood. It returns to normal size if training stops.

You'll never reach a state of hypertrophic cardiomyopathy (the bad kind of hypertrophy) with exercise. Its cause is usually genetic.

That additional oxygen needs to come from somewhere. Endurance training at the same time trains the heart to deliver more oxygen to the periphery; the primary mechanism is increased cardiac stroke volume.

Cardiac hypertrophy isn't a "bad thing". This is completely contextual. What you don't want, for example, is pathological hypertrophy from things like hypertension, or exclusive left ventricular hypertrophy without associated increase in chamber size.

The heart is very complex. You 100% should exercise it.

Sources?