Rust is a better choice, and it's designed with this in mind. It's still young, though, and I think there might be some as-yet-unsolved issues (these are things I've vaguely heard of and could be totally off-base) like binary size, ease of dealing with raw pointers, etc.
If I was doing this sort of programming for a personal project, I'd probably try using Rust, because I like it.
Dunno about Swift, though IIRC the current reference implementation may also currently rely on GC.