I've been trying to push things in that direction with my LLM tool - the idea is to have Python plugins which you can install that do all of the irritating details to get a model setup.
I've not yet been able to solve the challenge of needing CUDA etc for some models though!
Plugins so far: https://llm.datasette.io/en/stable/plugins/directory.html