When used sparingly and thoughtfully, I've never personally had an issue with it.
For example, I run shells in Emacs and have had to tweak loads of shell scripts written by colleagues to fix their poorly-implemented colourisation. It's useful to know when a test has failed; it's not so useful to have the whole terminal set to white text on a pale pink background.
One day I couldn't SSH into our servers from Emacs. It turned out somebody had edited .bashrc for the admin user to make the bash prompt blue. Emacs' TRAMP process was looking for a prompt ending in "$" or "#", not "$\[\033[0m\]", so it didn't realise the connections were successful.
There are two ways of handling this: we can blame the source of the bug (the person adding the colours incorrectly, or the assumption-loaded TRAMP regex), but there will always be more bugs in situations we'd never think of. Alternatively, we can avoid being 'too clever', and instead aim for consistency and least surprise.
(Actually, if you are suggesting that, I'm not going to disagree. But I am going to say that if so, those rules don't apply in the case of colored prompts, because colored prompts are useful.)
Anything we add on top of that, eg. ANSI colour codes, will be useful to some but harmful to others. The tricky part is working out which of those categories the current user is in.
I'm sure it's possible, but you'd have to acquire a new pty and decide what termios settings you want. It's a nontrivial hack, I think.
I'm actually kind of surprised it's not in moreutils[1].
[1]: https://joeyh.name/code/moreutils/
EDIT: Hmm, maybe it's not possible. I can't figure out exactly how to do it, anyway.
EDIT again: Apparently `script` does this on Linux: http://monosnap.com/image/Qlig4CHmQgV9pxvSmndVUgMTU88Adz
EDIT again: Expect's `unbuffer` also does it: http://expect.sourceforge.net/example/unbuffer.man.html#toc
Daniel J. Bernstein wrote a "pty" package back around 1991 that did this. Version 4 of the package was published in 1992 to comp.sources.unix (volume 25 issues 127 to 135). It's still locatable on the World Wide Web.
Bernstein later updated this, around 1999, with a "ptyget" package that was more modular and that had the session management commands moved out of the toolset to elsewhere. The command from that package to do exactly what you describe is "ptybandage". There is also "ptyrun". Paul Jarc still publishes a fixed version of ptyget (that attempts to deal with the operating-system-specific pseudo-terminal device ioctls in the original) at http://code.dogmap.org./ptyget/ .
As a bonus feature for people who use source code, there are similar "ptybandage" and "ptyrun" scripts, for which you will need Laurent Bercot's execline tool (http://skarnet.org./software/execline/), in the source archive for the nosh package at http://homepage.ntlworld.com./jonathan.deboynepollard/Softwa... . These make use of the terminal-management tools in the nosh toolset.
With both of these, you should be able to run "ptybandage uses_colours_if_tty | less -R"
So this feature must actually be present in the shell and maybe it is. I’m no expert but maybe zsh already offers something like this?