In my experience, the house of cards is usually debugging, not building. I've had to do some pretty crazy debugging, stepping through deep call stacks frame by frame, keeping track of big data structures as they change along the way. This can take a huge amount of focus that would absolutely be ruined by even a 5 second conversation.

With my experience now..

I fully accept that programming is purely a mental task. I choose to think with others and then code in my own time. The tasks are now so short that there is literally no space for someone to interrupt.

Everything else is productivity theater.

I don't think the suggestion is to "never" be subjected to collaboration, but certainly there should be a balance between collaboration and deep work that shouldn't be interrupted if possible.