Native is a format, RTB is ad transactions and delivery. They're not mutually exclusive. If you mean an entirely custom format then that reach is offset by the higher production and impression costs, which results in less overall demand.
Liquidity becomes an issue, which is why Reddit is a good example. They also use adzerk and built their own custom self-serve ad network but they make very little money compared to similar traffic using standard programmatic demand.
