I think you answered it yourself - other search engines know which clicks result in the longest dwell on page (as they run analytics and redirect search result clicks through their own referral page).
They could optimize their results for best results, but they don't want to, as that would reduce ad revenue.
While it sounds very conspiratorial, I don't think many teams in big ad/search companies are being measured by the lack of use of the product. For search, customer satisfaction is likely a first page answer resulting in no more searches on the topic (i.e. less use). For a company hosting ads in search results pages, that's a negative outcome. But it's what users want!
I believe if minds were put to it, and commercial drivers put aside, Google or bing or anyone else could very quickly deal with the scourge of poor results and SEO spam. But when you get revenue from search results page impressions, ad clicks, and impressions on the junk sites in listings, the incentives just don't align with the user.
