So, my imagination is lacking, so let me make sure I've got this right: The main gravitational perturbation you're thinking of isn't interaction with other small satellites or Earth's varying density; a bigger effect is going to be because the Moon, which is also moving relative to everything else is also a part of the equation.
Out of curiosity, wouldn't atmospheric drag have a circularizing effect on eccentric orbits? I'd have guessed that, since drag is greatest when the satellite is at its perigee, it's going to have a much larger relative effect on the apogee.
