For me, and from a mechanical perspective, the only truly difficult part of learning Japanese was kanji. The really annoying part is that Japanese doesn't use spaces and instead relies on transitions between the three writing systems to separate words, so even if you can get a hold of a page all in hiragana, it'll look like a wall of text. I still don't recognize much kanji, so for the most part I can only understand written Japanese after glossing it in WWWJDIC [0]. Furigana is a life saver.

Spoken Japanese, though, is easy. Well, at least from a mechanical standpoint. Japanese grammar is dirt simple and minimalistic, and I found it shockingly easy to learn. However, as someone else mentioned elsewhere in the post, Japanese is very big on inferring as much as possible from context and only speaking what can't be inferred. It can honestly be a chore just to determine whether a sentence is in the first person, the second person, or the third person, because Japanese is so aggressively pro-drop that most of the time, that information has to be inferred from context. And if you intend on speaking Japanese yourself, there's a lot of subtlety when it comes to picking the appropriate register to speak in. Use the wrong verb endings or the wrong pronouns (in this case, I'm glad Japanese is so aggressively pro-drop), and you'll stick your foot in your mouth.

[0] Most people use it as a dictionary, but it also has a really awesome glossing mode: http://www.edrdg.org/cgi-bin/wwwjdic/wwwjdic?9T