Kafka's a good choice. Newberry Award picks in children's literature have been a personal favourite. I'd probably find a place for Ursula K. Le Guin, Madeleine L'Engle, and Douglas Adams (very much in the spirit of Jonathan Swift IMO, and still underappreciated as such).

I've had the experience of trying to keep a friend well-stocked in audiobooks, and have made something of a practice of seeking out "best of" lists (best short-stories of the year, best books of the year, etc.), and ... find that there's not a whole lot that shows up in any decade that's especially good. Their own tastes tend to mid-century, relatively classical, and literary, and tends to discount themes increasingly prevalent in post-1960s literature (I feel the exposure would do good, but we're talking preferences here). Literary awards, "best books of" anthologies, etc., tend to improve the pickings but remain slim.

And again, financial motivation isn't helping, and by promoting far more low-quality literature, further clouds the field. For books --- big, solid, meaty, information-dense objects that take hours or days or weeks to assimilate, quality assessment itself is difficult. And financial motive, in authoring, publishing (cultivating authors, commissioning works, encouraging production, editing and rewriting assistance, packaging, marketing, and promotion) don't help the process.

Schopenhauer's argument isn't that most authors are financially motivated. It's that financial motivation leads to worse writing.

Again, you're focusing on anecdotes and "most authors" rather than the industry's own revenue focus. I find both uncompelling.

There are of course legions of writers (of books, of music) ... and other creators (art, photography, etc.) who do chase that dollar. Back in the day, Writers' Market was full of all the standard encouragement and secrets-of-the-trade for breaking through. That same advice is now much more scattered, but you'll find it online, much in the form of YouTube videos on storyboarding, either generally, or using writing tools (Scrivener seems popular) specifically oriented for that task. The objective is to quickly create cookie-cutter literature that fits a market's wants and needs, not creation of great literature.

Typical current advice (there are many video results):

Storyboarding generally: https://youtube.com/watch?v=JGeVXafMkwM

Scrivener: https://youtube.com/watch?v=AJyGox2ldHo

In the research world, it's grants-chasing.

The issue is that creative media (print, visual, video, music, etc.) follow power laws and tent-pole effects. There are a few big hits, there are an awful lot of also-rans. Ironically, the more global the market, the fewer winners (rather than numerous top-ten contents, there is only one --- any practice based on cardinality, that is, ranking, is inherently zero-sum. One of the better treatments of this I've found is in Charles Perrow's Complex Organizations (1972, 1979, 1986) (https://www.worldcat.org/title/complex-organizations-a-criti...), in the chapter addressing the music industry. Interestingly, its discussion of hit-making, labels, talent, backing performers, and corruption-dependent distribution systems (radio payola and the like) has eerily strong similarities with the tech sector's VC, founders, tech talent, and overly-credulous tech media (and lately, mobile-device app markets). There's a powerful lesson for HN's audience here.