Phonologically, Japanese is easier. No tones, and no retroflex consonants to worry about as in Mandarin (unless you don't mind sounding southern/Taiwanese). There is a pitch accent in Japanese, but apparently it varies wildly from one end of the country to the other, and it's not generally critical to comprehension, unlike Mandarin tones.
Lexically, Japanese is easier, at least IME. Don't know a word? Given all the borrowing in Japanese, I've had very good luck just to mangle the English word into Japanese phonotactics, and 90% of the time, the person I've spoken to immediately knows what I mean.
Syntactically, there's a lot more similarity in Mandarin's basic SVO word order to English.
But Kanji. OMG. You need to learn roughly the same set of Kanji as hanzi to be high-school literate, but the Japanese set will generally have a minimum of two pronunciations per character, which vary wildly in context. Mandarin generally has one pronunciation per character, which becomes far simpler to learn.
But, there's an important distinction. Japanese is singled out as "somewhat more difficult for native English speakers to learn than other languages in the same category."
Those are a mixture of my impressions, and the opinions of friends who speak the language (both learned and native). Overall, I think the most difficult part of mastering Chinese and Japanese is the same: Lack of shared cultural context, if you come from an English-speaking or European country.