Is that really true?
There are many examples of people who came to mathematics late in life and did good things:
http://mathoverflow.net/questions/3591/mathematicians-who-we...
I think you only get better with time if you keep yourself sharp. What has changed for me is that I can't sit still and pound back Redbulls for six-hour coding sessions and then hit the gym. My process is also more methodical.
But mathematics is certainly one area I've only improved in with time.