Did you compare it to a roughly equivalent Objective-C program? The top four entries in your profile data show retain, release and objc_msgSend. You will have those in Objective-C too. Maybe to a different degree? That's also why I am asking whether you have similar Objective-C code to test. Or maybe it's just that unoptimized Swift code is slower and optimized code is faster?

Compile times are a related but different matter. There are -warn-long-function-bodies and recently -warn-long-expression-type-checking which are really helpful and can give you an idea where most of the compile time is spent. In my experience, the type checker can spend a lot of time in mildly complex expressions involving overload resolution, which can be really annoying but there are often ways around it. With those culprits being eliminated, I have never encountered 5 minute build times for projects of this size or bigger, and I like to imagine that I write fairly generic code.