That's fair, but arguable. At this stage I think the architecture and it's reason to be are now so deeply intertwined that you cannot separate them (or would ruin it with the attempt).
I've seen instances where a building is all about what takes place there, demolished, re-built, and lose their soul and former purpose.
I can easily see that happening with the Southbank. It's iconic and embedded so deeply at this stage, despite the the various opinions on its looks.