I agree with everything you've said regarding cultural assimilation when you can't speak the language. If you're a knowledge worth in a major city, then you can get by since people respect you and your peers can speak some level of Japanese. But as unskilled labor life would be very difficult.