While they do swap more often than one with more RAM (obviously), at least with the M1s, the SSDs are stupidly fast, to the point that you barely notice it for day-to-day work. They nerfed the SSD on M2 base models; not sure about M3.
For the record, on my base M1 Air, I generally have Safari, Spotify, and Alacritty open at any given time. I can also run Docker Desktop if necessary, although I prefer colima since I don’t need any of the bloat.