> But they don't suck because of lack of effort - they suck because there are real physical constraints.

They do suck due to lack of effort or investment. FANG companies have remote dev experiences that are decent - or even great - because they invest obscene amounts into dev tooling.

There physical constraints on the flipside: especially for gigantic codebases or datasets that don't fit on dev laptops or have need lower latencies to other services in the DC.

Added bonus: smaller attack surface area for adversaries who want to gain access to your code.

> even on a VERY good connection, RTT on the network is going to exceed your frame latency for a computer sitting in front of you (before we even get into the latency of the actual frame rendering of that remote computer). There's just not a solution for "make the light go faster".

Are you imagining the implementation as some kind of Remote Desktop setup, where no software runs on the local machine (except the Remote Desktop, of course)? This is not the state of the art for using remote developer machines: Typically some editor/IDE components run locally, for example.

> Shared resources are unpredictable.

Then don’t share them! We should do something akin to physically moving your local computer into the data center next to the servers.

> Is my code running slowly right now because I just introduced an issue, or is it because I'm sharing an env and my neighbor just ate 99% of the CPU/IO, or my network provider has picked a different route and my latency just went up 500ms?

If the software you’re developing is going to run in a shared environment then it’s better you experience these issues while developing otherwise you’re asking for a lot of “works on my machine” problems.

> And that's before we even touch the "My machine is down/unreachable

This seems less about remote and more about stability/software design?

> I don't know why and I have no visibility into resolving the issue…

This seems more about observability/software design?

> We're not talking about machines running production data.

The source code itself is something that a company should protect for multiple reasons, not the least of which is preventing an attacker from reading your source to find exploits. There are also various legal and compliance reasons for limiting the distribution of source as much as possible.

> What I do want is a fast system that I control so that I can actually tweak it as needed to develop and debug the system

I don’t understand why this is impossible on a remote machine. Can you elaborate?

> it is almost impossible to give a developer "the least access needed" to do development locally because if you know what that access was you wouldn't be developing still.

I’m sure we can imagine all sorts of setups, from free-for-all root access to so locked down it’s impossible to do work. The sweet spot is typically “you can sudo within reason but we’re logging your activity.”