I don't know D, so I wouldn't know about it's downsides, besides the one you have spoken of already: lack of adoption. What are the downsides of D that wouldn't go away if it was widely adopted?
You say D is sufficiently old, that it's lack of adoption is significant evidence that we should never adopt it. The problem is, most people think like you do: they wouldn't adopt a language before it has some visible traction. The corollary is, new languages never get widely adopted, unless of course some Big Corporation does massive publicity around it (Java, C#…). The real reason D isn't widely adopted, is not because it's not better than C++. It's because it was not widely adopted in the first place. Textbook chicken and egg.
You speak of benefits and downsides of a language, but what you are really speaking of is the whole ecosystem as it is now. You don't even look at the long term potential.
Also, put yourself in the shoes of a language designer. Say you have a brilliant programming language. Experimental quantitative research has shown with no ambiguity that your language is better than some established, old language in its own niche (20% more productivity, 10% less post-release bugs, same observed runtime speed and memory consumption…). Just one little problem: your language have zero users, and only a little standard library. How would you get people to adopt your new language?
Hint: people won't believe your peer-reviewed experiments. People will refuse to learn a new syntax. People will want to reuse their old code. Even if you come up with an automatic translation tool, people will not trust it. People will say the benefits of the language are outmatched by the drawbacks of its standard library (too small), and of its community (none to speak of). People will be afraid of the required training (even if your research shows productivity benefits after a couple weeks).
How do you fight Status-quo Bias?
Personally, I found only one answer, which unfortunately isn't applicable everywhere: making the new language's implementation so cheap that all it's costs are paid off in a single project. DSLs may do that: with the right tools, a small external DSL can take no more than a couple hundred lines to implement.
