I am sure there was a whole bunch of factors, but, at least from my lay perspective, it seems like a recurring theme was that these companies kind of underestimated how good even relatively cheap Intel CPUs were going to get in the 90's, making it so that even consumers could afford a pretty powerful computer, or at least powerful enough to be "useful", and then it becomes less obvious why you'd spend $5,000-15,000+ on a fancy workstation; if I can get 80% of the value of an SGI with just a decent Pentium and a bit of extra RAM, for 1/4 the cost, most people are going to go with that.
Of course, I don't know what I'm talking about, I'm confident a lot of people on HN know more about this than I do, this is just a Wikipedia-level understanding of this seems to indicate to me.
Still, I do like to think about it. A part of me thinks, and I have no way to confirm this, that there might have been bigger ambitions for the N64, to convert it into a "real" computer, so you could have a "real" SGI machine at home (though obviously less powerful than an Indy or something).
