Of course it will limit who will be able to interact with your project, which can be a good thing or a bad thing. For the math-unicode in numerical software, you may not even want someone without at least a minimal math background (enough to understand Greek letters and math symbols) working on the code. Likewise, a project that’s inherently Chinese, where it doesn’t make sense for the users/developers not to know Chinese, should feel absolutely free to use Chinese symbols in their source code.

On the other hand, if you do it gratuitously, you just unnecessarily limit collaboration. I’m ok with that, too, personally: it’s your own foot you’re shooting.