Oh, yes, they were most definitively competing with Yahoo too, since Yahoo was the first port of call for non-technical users. Altavista was too complex.
For me, the speed and lack of ads vs. Altavista was irrelevant. When my friends and I finally switched, it was because the results finally convinced us that spending a lot of time tweaking search queries on Altavista wasn't worth it when Google would give better results on the first query. That realization was a long time coming.
Altavista's basic ranking was abysmal, and if you didn't notice any improvement it is probably because you like me knew how to search on Altavista. Regular users didn't. It worked well for technical users because we'd expect to do stuff like add "+term" and "-term" and quote phrases and use and and or. Regular users didn't.
I remember a lot of very long back and forth discussions with people I knew about whether or not Google would have a chance, because the above described method worked so well and it seemed so simple to use that it was hard for us to accept that getting non-technical users to use it wouldn't fly. Many of us were convinced that the Altavista model would win out in the long term, and it frustrated some of us (me included) to no end that Google were so stingy with their operator support even after we'd switched.
But as the net got flooded with relative beginners who'd type in a single word or two and expect decent results immediately, it rapidly became obvious that Altavista was a dinosaur.
Today we see just how much we (and Altavista) overestimated a large part of the user base:
Go to search.yahoo.com. Type "google" in the search box. Watch the second search box appear in the results with a "Get straight to your answers here with Yahoo! Search" above it... It's there for a reason: People do indeed search for search engines by name on Yahoo. I first learned of this one when I worked at Yahoo; they had numbers to back it up.