Stats/calc/algorithms weren't hard, I needed practice. I did every question in every textbook and followed up with the prof when I didn't understand how something worked.
Computer science actually pulls together all these different disciplines into one. By that metric, it might be the hardest. But really, nothing is easy or hard, it just takes practice.
As someone who had significant exposure to the world of mathematics for a working professional (4 years in an excellent PhD program) and who codes for a living, I find that what I did in grad school was multiple levels harder than what I do now, or what I get exposed to in the inane CS-focused interviews of some companies. The abstraction level is seldom, if ever, matched.
What about if you have to derive and prove the correctness of the formulas used instead of just memorizing formulas without background?
Funny how both of us are dismissing stuff neither of us did in school :)
I should note that I did study a year in California. There, the computer science was much harder, and the math was much easier (scored straight A's in math courses literally without any effort, and nearly failed the operating systems course that I took).
We should probably think of Computer Science in the same way. Instead of just vanilla CS, we could split a CS BA/BS into just as many specific concentrates: assembly, architectures, crypto, data related anything (management, safety, security, databases), etc.
But they don't have to learn it all in school. The way for CS to cut itself down to a manageable size is to concentrate on the traditional "hard core". Compilers, operating systems, plus some newer (i.e. from-the-1970s) stuff like networks, distributed systems and concurrency.
As always, people will cry that they never (ha!) do such stuff in their real jobs. But the more day-to-day stuff can be learned on the job. The stuff I listed is where university-style learning can offer a real advantage.