Arigatou, literally, means "difficult to be/have" (有り難う (arigatou) is an euphonic change (onbin/音便) of 有り難い (arigatai; ari/aru: be/have...; ~gatai: difficult), which translated like this is just completely weird, but so is literal translation.
Arigatou is fine in many more cases than you claim it to be, but yes, Sumimasen can be appropriate in many cases where you would say "Thank you" in english.