That's because the illegal action consists of being in a location where you have no right to be. And it's not, in fact, the only case. Compare it to its close relative, trespassing:
An "illegal resident" is someone living in a country of which they have neither citizenship nor visa.
An "illegal occupant" is someone living in a building of which they have neither possession nor tenancy.
It seems rather straightforward to me. Whether such a view is "un-American" or not I legitimately do not know, not being an American myself. Though I can say that that word doesn't have the proudest of histories, to put it mildly.
