That's the only way I can see it working, and I'd be interested in something like that, but I am dreading the technical support system needed to handle all the issues like CEC not working, IR codes (if you go that direction), shitty TV firmwares, passthroughs, various amounts of audio delay (if you plan to switch between on-device audio and TV's audio), and all kinds of other stuff.

But I would be interested in a device that could pull it off. Doubly so if they were cheap enough to get one for each TV.

Of course, but if you are going to try and make a product which relies on well working and fast "smart" (or otherwise) TVs and there aren't any (or enough) of them today, your product isn't going to get very far.