Better primality testing would be interesting to the crypto community, but it could rock the math world - like "Primes is in P" introducing AKS as the first deterministic polynomial time primality test.
As you said, though, better factoring would be game-changing for both cryptography AND mathematics.