Hence this announcement is great, since it seems to say they’re (going to?) support building GUI apps with SwiftPM and/or the newly open sourced build tool.

Building Swift code for iOS without going through xcodebuild is sort of obnoxious but is possible. You do need to have a copy of Xcode installed regardless of programming language simply because the iOS SDKs aren't distributed separately.