Let's say the first hour is 100% productive, the 2nd is 60%. Perhaps the third is 30% and the 4th 20%? But would they really undo the good of hours one and two?
The research quoted in a lot of the Deliberate Practice literature suggests we can only focus intensely for 4 hours a day. Then it's a question of wasted time, or harmful time.
So, I do think there is a limit for consciously doing hard math. Subconsciously, however, who knows what one's brain is doing? There's plenty of anecdotes about working hard on a problem for hours without apparent progress and then, suddenly, having the breakthrough insight during a walk or in bed, supposedly during a break of working on the problem. Famous anecdotes are Archimedes in bath and Kekule's dream about snakes and benzene.
Now, chances are these guys were still thinking of the problem (one advantage of theoretical work is that you can combine it with most low-effort activities) and nobody who gets such an epiphany knows whether just keeping churning would have led to the same result, possibly earlier, but I think that there is some truth in this. Just as running for 16 hours a day is not the best training for any race, it is good to have breaks from doing extensive math.
Also, in my experience, real understanding of mathematical concepts comes to me after I've studied it, when I'm thinking about the matter in the background while doing other things. It's plausible that, if I were to overextend myself, I'd lose focus and not assimilate that much later.
