Isn't there already an overlap in the upper 90% between humans and apes? I don't know how much the neanderthal DNA differed back then, but it couldn't be more than that, could it? So wouldn't 3% of the total be at least a third of the parts that did differ?

There are a number of ethnic groups (not species or subspecies, I realize) that are less than 3% of the gene pool today (and happened over a much shorter timespan I would suppose) such as Irish, Jewish, Armenian, etc. Would they be considered having been outcompeted at this point?

I think that's the crux of the cognitive dissonance we all experience when described the trope for the first time. 3% of group B's genes persisting eons later in the combined genome A+B isn't necessarily reflective of the original relative size of the population of B vs. A. Evolution only countenances which genes conferred a survival benefit edge to persist. Suppose population A was one million individuals, and population B was only one hundred (a 10000X disparity). Assume complete absorption of B into A over an instantaneous period of time. The proportional genetic representation argument, if it were operative, would imply that eons later, only 0.01% of the combined genome is from population B. Only that's not how genetics works - what matters is how much of a survival benefit did population B's genes confer on the inheriting offspring? To put up a concrete example, if population B had a gene conferring immunity to a regionally endemic pathogen, then that pop. B immunity gene is going to quickly saturate representation in the offspring populations as pathogen-vulnerable population members die off from disease.