In general it appears that it's just roads that aren't publicly accessible which one can't go down. It's not just Downing Street, for example, but a bunch of pedestrianised places, presumably for the obvious reason that the Street View car couldn't go down them.