The Nazis did plan an invasion. See Operation Tannenbaum[0].
Switzerland didn't end up getting divided between Germany and Italy thanks to the Red Army hurtling more humans west than the Wehrmacht could supply bullets east. Swiss munitions production wouldn't be risked.
[0] https://en.wikipedia.org/wiki/Operation_Tannenbaum