If you're born Japanese then you lost the language lottery. Japanese has a limited set of sounds and there are very few close languages grammatically; Korean being the major one. So even the global language, English, is a massive challenge. It's easier for a Chinese person to learn Japanese than the other way round. (Japanese speakers at least have a big leg up on reading Chinese)
I wonder if it's easier to go from Chinese to English, they certainly have more sounds/tones an English speaker will have never spoken. In reality, major different is probably a power difference, it's a lot more useful to know English than it is to know Mandarin in the general case. (If a Chinese student goes to France they'll be using English)