1. As you say, the introductory classes are enormous. For the math classes this means less feedback on your assignments. What this also means is that you have other undergrads grading your work. If that doesn't terrify you, then it should.
2. Inadequate preparation. Similar classes in other departments have more prerequisites. I'm currently attending a machine learning class where more than half the class is probability-illiterate. Take a look at how the core math curriculum is taught (algebra, analysis)
3. Professors and graduate students are out of sync about the material and assignments, which leads to situations where nobody can answer your questions. See cs162, cs61c. I've heard similar rumors about 161.
4. Only 5-10 upper division undergraduate cs classes. Interested in type theory? Practical encryption? Computer algebra? You're out of luck. Most other universities seem to offer a lot of variety.
5. Less treatment of the fundamentals. Our systems class no longer goes into any depth about malloc. They've replaced it with a section on map reduce. Similar situation with the operating systems class and I believe the security class.
6. The people who rise to the top are the people who were probably going to be successful anyway. That's just my experience/bias though.