The only obvious downside to an outside observer is that they will find it difficult to collect 'all weather' data in substantial volume in less than approximately a single year.

From the positive side, they will collect and be able to evaluate a much /wider/ range of driving data from a far larger set. They are much more likely to run across 'better morons' both in operators of the Tesla and in operators of other cars.

The question whether a time average over a long period of a random quantity in one system and an average over multiple systems over a short period are the same is the question for ergodicity [1]. If both are the same, the random process or system is called ergodic.

The answer is: If the measurement period for the average over multiple systems is long enough that each random variable has visited a representative sample of states which make up the majority of contributions to the mean value, then the system is ergodic.

[1] https://en.wikipedia.org/wiki/Ergodicity

Here's an interesting problem I hadn't thought about before. Teslas are expensive and rich people drive in different places than poor people, so teslas will have more data about places rich people go. That's another interesting data imbalance.