In fact there are an infinite number of correct Rust programs which will never access memory incorrectly but which will still be rejected by the compiler, for the simple reason that Rust's authors, talented though they may be, have made no progress at all at solving the Halting Problem.
What Rust actually accepts is a subset of correct programs. And this subset is somewhat informally defined as "whatever the compiler could manage to prove". Real code hits against this limit occasionally, and when it does there's really no option other than "join the Rust team" or "try voodoo".