Different branches of mathematics develop their own notation, so there isn’t _a_reference_, there _are_references_, just like there are for programming languages.
Also like with programming languages, there are communalities that are valid most of the time. Simple examples that you probably know are the way we write numbers, using + for addition, using {} to denote sets, ℝ for the real numbers, etc.
You’ll find that many math books start with an overview of the notation used.
https://en.wikipedia.org/wiki/History_of_mathematical_notati... mentions tons of specialized notations (most of which, I guess, you won’t need to know about)