Ask HN: What it takes to build Jarvis using current state of the art LLMs?

For people who don't know Jarvis, it's a know-it-all personal assistant which just doesn't answer but also perform actions based on commands. It's catered to specific person needs. We want this to run on your preferred device instead of cloud

I thought we were miles away from building something like this because of so many complex parts involved

1. State of the art speech to text generation ( Shout out to Whisper here )

2. State of the art question answering ( GPT-*, LLama, etc..)

3. Integration with your virtual life ( emails, music, finances etc..)

1 & 2 have definitely shown significant progress. the major bottle neck right now is 3. Most of our virtual life is centralised and no one provides a clean API such that we can actually have a holistic view of everyone's digital persona in one place. And last but not least, privacy and security is super important considering we are giving the total access to a bot. Am I missing anything else? Would like to see some healthy discussion on this topic