The M2 is a lineup of chips with a range of specs (much like i3, i5, i7, i9 lineups). The M2 that powers the iPad is 1/3 the performance of the M2 in my MBP. I suspect the Vision Pro's M2 will be somewhere in the middle to manage heat and battery life. While the Vision Pro M2 can technically "run it", it will still run faster on a dedicated external machine that can handle the heat and power better.
Add in the need to render a very large virtual space and you end up with a lot of capacity being used by default. That's likely fine for "iPad like tasks" (web-browsing, email, messaging), but it seems insufficient for anything requiring processing or memory.