My Mac isn't an M1, but it does only have 16GB of RAM. Yes, it goes into swap when Safari grows like that. It's mostly ok even with several GB of swap used, but can slow down.
My solution is to restart Safari when it gets too bad, as it's obviously leaky.
For a long time I used Firefox, was annoyed at how slow it would get on a busy browsing day, and didn't realise the memory consumption of Safari (also open) was overloading the poor machine. One day I saw the stats and realised what was happening. Now I open only one browser at a time, and everything is much nicer.
If I decide to get another Mac (undecided), I'm holding out for an M1X or whatever with more RAM. 16GB isn't comfortable for my work any more. I'm not the kind of person who casually buys new expensive machines, so won't be getting the x86 32GB as an intermediate knowing I don't really need it, as I think it would be better to end up with both an x86 (which I already have) and an ARM going forward. I'm into code generation and portability, so that's better for me. And I like the idea of less fan noise!