And most applied maths doesn't seem to worry about proofs much. They have techniques that either work pretty well or blow up.

My intuition is that a regular LLM is better att coming up with a correct task description from a fuzzy description than it is at actually solving tasks.

There's a lot of automated proof checkers out there. Presumably you would just run any solution from an AI through those.