IIRC, there were a few different reasons why it was scary.

One of the first was this mission was the last one, it was a difficult repair—SC4 did the first in-orbit repair—to do at the time, especially outside of the telescope. Because when you leave the safety of the shuttle, you’re traveling at a rate of ~17,000MPH and where the hubble orbits, there is a much higher chance of debris floating around than as if they were lower in the atmosphere. So in laymen terms, you’re attached to a cord that is supplying you with O2 and spinning at 17,000 MPH while trying to avoid rocks and other dangerous floating objects.

One other danger was the O2 levels. He said the suits are heavy, all of your tools are attached to you via cords and you have limited time to be outside. If something where to happen in this situation, not only would you become unconscious within seconds from the lack of O2, you’d pretty much be a forever piece of debris floating around the galaxies as your body will never decompose or fall from orbit.

The latter part was one of his biggest worries. He knew that he’s on limited time, he also had to renter the shuttle slow because it’s similar to divers and decompression. If for some reason he went too fast or there was a leak somewhere, his suit would have instantly filled up and the pressure would have blew him apart inside the suit.