It is possible, but difficult. You may lose access to some groups, but you can't have everything you want without some sacrifice.
Personally, I'm leaving WhatsApp. Yes, my family and friends will be a bit annoyed about the hassle of contacting me separately, but so be it.