undefined | Better HN

0 pointskohlschuetter1mo ago0 comments

I was hoping somebody would suggest that (and eventually try it out).

With TB5, and deep pockets, you might probably also benchmark it against a setup with dedicated TB5 enclosures (e.g., Mercury Helios 5S).

TB5 has PCIe 4.0 x4 instead of PCIe 3.0 x4 -- that should give you 50 GbE half-duplex instead of 25 GbE. You would need a different network card though (ConnectX-5, for example).

Pragmatically though, you could also aggregate (bond) multiple 25 GbE network card ports (with Mac Studio, you have up to 6 Thunderbolt buses, so more than enough to saturate a 100GbE connection).

0 comments

rbanffy1mo ago

Too bad Jeff Geerling returned his Mac Studios to Apple. Would be lovely to see how 5x faster RDMA impacts the performance.

geerlingguy1mo ago

25 Gbps isn't all that much. It would be good, but would be below the 40+ Gbps I was getting on the TB5 ring network.

I think where it would show more significant speed up is on the AMD Strix Halo cluster.

Except I haven't been able to get RDMA over Thunderbolt on there to work, so it'd be apples to oranges comparing ConnectX to Thunderbolt on Linux.

pdrayton1mo ago

Oddly enough, that’s exactly what I’ve been benchmarking - different ways of linking Strix Halo machines - with respect to throughput & latency.

Posted a little bit re: the TB side of things on the Framework and Level1Techs forums but haven’t pulled everything together yet because the higher-speed Ethernet and Infiniband data is still being collected.

So far my observations re: TB is that, on Strix Halo specifically, while latency can be excellent there seem to be some limits on throughput. My tests cap out at ~11Gbps unidir (Tx|Rx), ~22Gbps bidi (Tx+Rx). Which is wierd because the USB4 ports are advertised at 40Gbps bidi, the links report as 2x20Gbs, and are stable with no errors/flapping - so not a cabling problem.

The issue seems rather specific to TB networking on Strix Halo using the USB4 links between machines.

Emphasis to exclude common exceptions - other platforms eg Intel users getting well over 20Gbps; other mini PCs eg MS-1 Max USB4v2; local network eg I’ve measured loopback >100Gbps; or external storage where folk are seeing 18Gbps+ / numbers that align with their devices.

Emd goal is to get hard data on all reasonably achievable link types. Already have data on TB & lower-speed Ethernet (switched & P2P), currently doing setup & tuning on some Mellanox cards to collect data for higher-speed Ethernet and IB. P2P-only for now; 100GbE switching is becoming mainstream but IB switches are still rather nutty.

Happy to collaborate with any other folk interested in this topic. Reach out to (username at pm dot me).

j / k navigate · click thread line to collapse