For one thing, at the time, Unix and all its associated tools were really not prevalent outside of Bell Labs and a few places — troff would not have been available to run on the computer systems Knuth had available to him in 1977. (The ones at SAIL, which were PDP-10 machines running either TOPS-10 or WAITS, I forget what I read.) He didn't have the source code of troff either; all he had was published descriptions: and those he did use, e.g. the paper by Kernighan and Cherry describing `eqn` came out in 1975 and he used ideas from it while designing TeX's syntax for math.

For another thing, troff was designed to control line printers and other such devices that were typically attached to computers; it simply wasn't designed to drive a digital typesetter of the kind Knuth was thinking of (Alphatype CRS). So he'd need a new program anyway. In fact, he says that until he saw Patrick Winston's book on AI in 1977 (which was one of the early books produced on a digital typesetter, i.e. where the actual technology for getting ink onto the page was based on 0s and 1s, instead of being based on either pieces of lead (hot metal typesetting) or light and lenses (phototypesetting)), he had simply not associated output from computers with “real books”. Things like troff were designed to control the simplistic devices that came with computers, that could move by discrete (large!) amounts on the paper (e.g. some could move by full “lines” only), and strike down specific symbols (from a limited font repertoire that came with the device) onto the page. This wouldn't have seemed relevant to Knuth, who primarily wanted to create his own digital fonts (as he did with METAFONT), and TeX was among other things simply a way to use those fonts.

So, regardless of whether one considers troff to have very beautiful typesetting or not, it simply wouldn't have seemed applicable to the problem at the time, to be rejected as "not good enough" or not. Now of course with output going to generic page-description formats like PDF (even TeX's DVI was not Knuth's idea—he was thinking TeX would just write code for the device directly—and he had to be convinced to do it by Fuchs: https://tug.org/interviews/fuchs.html), and high-resolution printers (fairly high DPI, even if not 5000+ as with the CRS then) available to typical consumers, we don't appreciate all this. :-)