it's all just driver/options crap if I were to take a guess.

there are a lot of usb options that matter, and tp-link ships lots of realtek chipsets that require very special driver incantations that a lot of the linux drivers simply don't replicate.

two+ layers of bad options will surely add 4ms quick.

I think there's definitely something with that specific setup. For me, pinging between two cheap Realtek 2.5 GbE USB dongles (one is on a Mac one is on a 7 year old Intel Atom Synology) is still sub-ms (hovering around 0.7-0.8ms) so it's not an inherent problem to USB dongles.

USB itself can have a lot of issues anywhere in the chain. I have a Thunderbolt dock where half of the USB ports adds latency and reduced throughput just because the USB chipset that powers them is terrible (it has two separate USB chipsets from different brands). Switch to a different port on the exact same dock and it's fine.