e: I also think, on reflection, that a signfigicant part of your ability to grok a new paper per your comments is your comfort in approaching these concepts due to your familiarity. Think of learning a new language - once you have a feel for it, you're likely more comfortable exploring new concepts within it, however when you're faced with it from the start you probably feel very lost and apprehensive.
I feel that understanding calculus is a fairly fundamental step in the 'language of maths', teaching that symbols don't necessarily represent numbers but can represent concepts (e.g. delta being change). This isn't something you encounter until then, but once you do you begin to understand the characters associated iwth integrals, matricies, etc. in a way that you may not have previously with algebra alone.