In the limit, I think most people would agree there are diminishing returns. It's better to spend that 20th hour each day sleeping than studying.

So, I do think there is a limit for consciously doing hard math. Subconsciously, however, who knows what one's brain is doing? There's plenty of anecdotes about working hard on a problem for hours without apparent progress and then, suddenly, having the breakthrough insight during a walk or in bed, supposedly during a break of working on the problem. Famous anecdotes are Archimedes in bath and Kekule's dream about snakes and benzene.

Now, chances are these guys were still thinking of the problem (one advantage of theoretical work is that you can combine it with most low-effort activities) and nobody who gets such an epiphany knows whether just keeping churning would have led to the same result, possibly earlier, but I think that there is some truth in this. Just as running for 16 hours a day is not the best training for any race, it is good to have breaks from doing extensive math.

It could, in mental work. You could forget things.

Also, in my experience, real understanding of mathematical concepts comes to me after I've studied it, when I'm thinking about the matter in the background while doing other things. It's plausible that, if I were to overextend myself, I'd lose focus and not assimilate that much later.

My unscientific gut feeling based on personal experience is that is harmful. After a few hours you need to step back and give your brain time to digest and store what is has studied. If you don't give it that time then not only won't you be able to learn new things, but you won't later be able to recall things you already studied.