I think most folks would agree that at some point Hitler's Third Reich could be opposed by force, but I think folks would struggle at exactly when to draw the "ok, it became OK to punch Hitler's brownshirts" line.
I don't think it as clear cut as you claim. There are many political and psychology thinkers who hold that extrajudicial violence is usually counter productive.