It matters a lot. I just spent three days on a 130kbps 1000+ms latency high packet loss connection, and RTT was one of the problems that made lots of sites I tried (practically) unusable, due to launching dependent queries in series without any retries. The other big one was gigantic interaction-blocking assets served from CDNs with aggressive timeouts.

SV techbros are a fun pinata to smash, but that's a load of bullshit intended to conflate individual workers (easily hateable and zero power to fix anything) with ad-tech/paywalls/lead acquisition/overoptimisation driven by profit-seeking (too diffuse to hate, too little understood for lawmakers to regulate effectively...yet), and make slow websites seem like a technical or competency problem instead of an auctioning-your-eyeballs-and-spamming-you-with-modals.