And the reason why not is simple. Anything that Microsoft thinks is a good thing to add to the API, they'll add for themselves. When the new API is released, their software is released with it. This gives them a competitive advantage over competitors who have to wait for Microsoft to have the idea that they want, and then scramble to implement it after Microsoft does.
The EU is suspicious of this for the simple reason that Microsoft has a several decade history of doing exactly that. Repeatedly. My favorite example being the release of Windows 95 with Microsoft Word available at the same time, and with WordPerfect unable to run. By the time WordPerfect had figured out how to port their software to Windows 95, they were no longer the market leader.
That is somewhat revisionist history. WordPerfect admitted at the time they saw OS/2 as the future and were focused on that. Only in hindsight did they realize OS/2 was going nowhere (too bad, it was better than 95) and had to rush to get a WordPerfect for 95. Worse for them, they wrote each release of WordPefect in platform specific code (mostly assembly) so it wasn't a case of port to 95 it was a case of start over mostly from scratch.
Yes WordPerfect lost to Word with 95 - but it was bad decisions on WordPerfect's part. They had opportunity to get WordPerfect on 95 much faster. I don't know if it would have been fast enough, but they didn't even try until it was too late.
The use of platform specific code was a performance necessity at the time, everyone did it. Part of the promise of Windows 95 was that it could run your Windows 3.1 programs. They bent over backwards for a ton of programs, but not WordPerfect. Microsoft also had an early access program to Windows 95. WordPerfect applied for it - and was denied access. After that the OS/2 bet was their only real hope.
The truth is that Microsoft had a long and documented history of using one monopoly to leverage into another. Over and over again they lost antitrust lawsuits, but internally regarded them as speeding tickets on the way to greater monopoly power. This history showed up in court. The internal documentation on the WordPerfect case showed up in the Netscape case, and is part of why Mocrosoft won.
It wasn't until the EU started charging Microsoft over $400 million per day for noncompliance in 2006 that Microsoft's attitude started to change. Now I see them as just normal big guys with a worse than average history. But back in the 90s and early 2000s? They EARNED the title of "evil empire".
It was full of ad infested solutions, which would crash your computer from time to time.
Defender at least was reasonably performant and tended to be stable.
You could say that since they had access to kernel source, they were better informed, but I guess if there was an API, the provided documentation would solve the issue (not necessarily, not everyone bothers to read the docs).
But then you get back on how to enforce equal and open access for everyone (the EU did try to make Microsoft open the Word file format, but turned out it was so complicated and documented in legacy code only, that Micorsoft had trouble giving useful docs)
Anyway, as you said, it's complicated...
If I write a new OS how will you force the "word processing, video conferencing, and music-selling" companies to write code for it? If they don't write the above my OS is worthless, but if my OS fails in the market anyway they just wasted a lot of money. This is why OS companies tend to have the other things, their OS cannot exist in a vacuum and the only way to ensure they have those needed tools is to write them themselves.
Nobody wants to try to be selling consumer software that is optimized for the out of date and unsupported version of the OS.
If you could prevail on a government to decide that, maybe it could work.
One thing I see, is that AV has a component of maintaining a DB of signatures of bad things. This does not seem at all the job of the core os. Would the Debian team maintain such a DB?
The ultimate lesson then is to stop using MS stuff.
The fact that Microsoft abandoned it as soon as a regulator pointed out how anti-competitive the design of the API was makes you wonder what Microsoft's true intention was. To me that implies the anti-competitive design was its main feature and to Microsoft it would've been pointless to continue without it.
But another way of looking at this would be that perhaps they wanted to be the beta testers of the API themselves because opening it up would have been a maintenance liability for the company. Microsoft tends to be pretty good about backwards compatibility in ways that Apple is not.
We also don't know that these APIs were cancelled, they may make it into future versions of windows.
