As a self-taught programmer, I think you're downplaying how much you can learn in a CS program. I don't pretend to know as much as a CS major. I haven't spent time learning compilers or DB internals, and probably haven't studied CS algorithms enough to pass an interview at Google or Facebook.
Regarding the CS curriculum, an undergrad EE curriculum is similarly shallow: it's a sampling of many different subtopics that often are only loosely related (E&M, DSP, circuits, control theory, devices, etc.). Undergrads won't be experts in any one of those fields.
Since I graduated (grad school) in EE, I've told myself each year that I'll jump from my EE job to a programming job, since I enjoy programming more. But it's getting harder and harder to justify. Will I be leaving a senior EE position for an entry-level software position? Does that make sense at 30? Why am I the oldest person in the office when I interview at a startup? Why is the interviewer fixating on compilers and database internals?
If I were to do it over, I'd get a degree in CS because that's the most direct route to the job, and education continues on the job. Everything else was a diversion. And you're seeing the grass as greener on the other side of the fence regarding engineering/physics curriculum. Learning how to solve physics problems doesn't make you smarter or a better programmer.
