I wrote a blog post[1] describing what a local only LLM could do. The answer is quite a lot with today's technology. The question is - do any of the tech giants actually want to build it?
The locally hosted scenarios are in some ways more powerful than what you can do with cloud hosted services, and honestly given that companies could charge customers for the inference hardware instead of paying to host, it would likely be a net win for everyone. Sadly companies are addicted to SaaS revenue and have forgotten how to make billions by selling actual things (with the exception of Apple).
[1] https://meanderingthoughts.hashnode.dev/lets-do-some-actual-...