But most HTML in the wild today is not valid XML. There is XHTML as mentioned by a sibling comment but it's rarely used. So if you were to start with an existing base of HTML documents, you couldn't easily add XSLT preprocessing to them. The issue is with the input rather than the output.
Here is what I know about using XHTML in practice: https://www.nayuki.io/page/practical-guide-to-xhtml
If you have someone else's documents, or need to mass convert your own to fix them, there's HTML tidy[0]. This one is quite useful to be able to run XML processing CLI tools on scraped web pages.
But the real power is in delivering XML to the client, not HTML. This lets you work in the domain model directly on the frontend, and use XSLT to transform that into HTML for display. So of course you'd use well-formed XML in that case.
Imagine if you didn't have a distinction between APIs and pages; you just returned the data along with a link to a template that says how to display it and (ideally) a link to a schema definition. Modifying templates for components could be as easy as modifying CSS attributes in the browser console, giving the next generation an easy way to peak and play around with how it all works as they're growing up. We were so close, and then things veered off into the hellworld that is the modern web.
Still, it's possible with XSLT 1.0 to produce documents in the common subset of XML and HTML5 ("XHTML5"). It can't produce the usual <!DOCTYPE html> at the top of the document, but it can produce the alternative <!DOCTYPE html SYSTEM "about:legacy-compat">.
On the input side, every XSLT version only accepts valid XML, as far as I am aware.
[0] https://www.w3.org/TR/xslt-xquery-serialization-30/#html-out...
As for output, the difference is largely irrelevant for browser purposes since they just want a tree.
I'm not sure how many extensions the browsers allow, but a major part of the reason XSLT 2/3 failed to take off is because libxslt already provides most of the useful features from newer versions as extensions (many via EXSLT-namespaced modules, at least partially supported in browsers - see MDN); what it doesn't do is implement the unnecessary complexity that the Java world loves.
