Well really it's about respecting the performers and your fellow attendees. I once saw Mitsuko Uchida storm off the stage in Southbank because there was an alarm outside going off non stop lol.
If you went to a nice restaurant and I came over and started talking loudly with spit flying into your dish or took some food from your plate it would ruin your enjoyment. When we're trying to hear the subtleties of interpretation from the performers, the faintest sound detracts from that.
I think most people can accept that talking in a cinema or using your phone is pretty anti social so I don't know why classical music gets such a bad rap for being pretentious when it has a conduct. The only thing that drives me crazy is people always seem to cough when the music slows down and gets quiet. Please do the exact opposite. Don't spoil those tranquil serene moments with your coughing. Cough when the percussion and the tubas are storming and no one will hear.
That said, I'm all for bawdy music experiences - I once saw a tiny production of Le Docteur Miracle in a bar in London which was extremely fun (by Popup Opera UK) and if you go to Sarastro's by Covent Garden you'll get some fun rowdy opera hits sung at you while you dine. Plenty of outdoor picnic concerts are similarly laid back