I agree with a lot of your points, but can you explain the part about it doing a complete rebuild every time? It doesn't do that for me (unless I specifically tell it to).
My apologies: I just wrote a basic HelloWorld.java and a build.xml to go with it, and it looks like it doesn't recompile the class unless there is a change to the source .java file. So I was mistaken about that.