Is there progress happening in that trajectory?
There was a recent Hackernews post which had a novel approach about making agents interact with GUI/computer-use
https://news.ycombinator.com/item?id=47125014: The First Fully General Computer Action Model : https://si.inc/posts/fdm1/
Hope this helps