Building Swift code for iOS without going through xcodebuild is sort of obnoxious but is possible. You do need to have a copy of Xcode installed regardless of programming language simply because the iOS SDKs aren't distributed separately.

Hence this announcement is great, since it seems to say they’re (going to?) support building GUI apps with SwiftPM and/or the newly open sourced build tool.