The rationale is, because I want to.
The system has evolved over the years, current configuration is: Several 1080p SONY cameras with hacked firmware that stream video to a capture device. An older view of the camera rig that has since evolved again, is here: https://youtu.be/dGRDB1vVxyY
Some 4K webcams connected over USB that I don't stream. I capture one full frame every X milliseconds.
Two Kinects set to be out of phase capturing the entirety of the office as a depth map.
Two Rode shotgun microphones capturing audio and feeding it in to a Focusrite box.
Custom built USB "keyboard" with a few arcade buttons that permit "pause/unpause", "forget a little bit" and "forget five minutes."
Two LED lights to indicate recording status for both myself and anybody walking in the office.
Timesnapper on Windows, and a little custom C++ capture program for macOS and Linux that takes a snapshot of my desktop every X seconds.
All that data gets stuffed on to a secured drive on a file server. The data goes back more than a decade. Nobody has access to that data but me.
I use an NVidia Jetson to analyze everything: the desktop images, build up a map of applications, analyze people in the room, identify who they are, what clothes they are wearing, identification of activity, OCR of images, transcription of spoken word to text, identification of websites, identification of music playing, "oh hey, he's listening to the following artist, let me pull that artist's social feed and put it on the ambient screen in the hallway", which is kinda creepy when the software identifies my own music https://soundcloud.com/justinrlloyd and then stalks me and puts up my own social feeds on the household ambient screen. I also have the Jetson watching the front door via the Ubiquiti doorbell camera and can switch on the TV in my office if someone comes to the front door so I can see who it is, and also will notify me that a package is on the doorstep ready to be brought inside via the second high viewpoint door camera performing a "what changed in this scene, is that a package? That looks like a package. Package! ZOMG! Package! Package!!!" That algorithm has one job and it does it really well. Like a hunting dog staring at squirrels.
Lots of this stuff is readily available as ML models, for the most part I just strung them together with simple scripts to move data around.
I have a "virtual assistant" that I wrote, using NLP and key phrases with a speech recognition model that understands specific commands and some free form speech, an early prototype of my virtual assistant is here: https://youtu.be/uhl8wN7Uvv8 and I state for the record that it has gotten far better in the intervening years. And then a text to speech model when absolutely necessary to give me voice prompts.
This virtual assistant can control cameras, e.g. tally lights, zoom and focus, recognize the fact I am holding a receipt from a grocery store, or a book, and take a high resolution picture and tag it with meta data.
I keep a near real-time backup of my computers, and that data goes back probably three decades, any time I retire a machine I take a full drive dump and store that.
Out of office, I take a snapshot of my desktop on the laptop (Microsoft Surface or Macbook Pro), which is then automatically copied to the server when I return to the office. I built my own Sensecam-like device using a J2ME device almost two decades ago, but have since moved to using an Autographer for life logging.
This is just amazing. You've basically built your own JARVIS, and I really want to know how much you would charge to replicate this ? ... its almost impossible to turn something so bespoke into a purchasable product, so I'll never be able to walk into a store and walk out with this entire thing, but its like art, you can pay the artist to make you one thats pretty similar since they know how to do it... I suppose that does open a more interesting question for the wider audience.
How much effort does it take to maintain this?
With regard to maintenance, I don't know, I haven't touched it in months. Maybe every once in a while I get some weird, catastrophic failure of "I can no longer talk to the ambient screen, but I am still sending commands" and then I have to spend a half-day debugging what the issue is only to realize that the CNC in the workshop stole the IP address because I forgot to lock that IP to the device's MAC when I switched out the old screen for a new screen.
