While at a higher level, thunderbolt and https://en.wikipedia.org/wiki/ExpEther can both of course work over fiber too!
(Q|O)SFP are basically just raw high speed serial interfaces to whatever - you see this a lot in FPGAs, you can use the QSFP interfaces for anything high speed - PCIe, SATA, HDMI…
> [snip]
> As explained in the intro, this doesn’t come without a host of compatibility issues, least of all PCIe device detection, side-channel clocking and for PCIe Gen 3 its equalization training feature that falls flat if you try to send it over an SFP link.
So, uh… what’s the benefit? How much overhead does Thunderbolt really introduce, given it solves these other issues?
- Learning : I want to learn about the lower level of PCIe and it's a good project. - Re-use of cabling : I have a bunch of single mode fiber bundle going around already. You can't find thunderbolt that just have a LC connector ... - Isolation : Active thunderbolt cable still often have copper for some low speed signals, they don't offer true galvanic isolation - Avoid dealing with thunderbolt. I want a custom chassis/pcb at one end and chips to convert from TB back to PCIe are not readily available to make custom stuff with ... (not as an individual anyway).
So yeah, if you want a ready to use solution, TB cable is absolutely a good choice, here I'm having some fun, learning in the process and hopefully sharing some of the knowledge.
Thunderbolt 5 offers 80Gbps of bidirectional bandwidth. PCIe 5.0 16x offers 1024Gbps of bidirectional bandwidth. This matters.
TB5 cables can only get so long whereas fiber can go much farther more easily. This means that in a data center type environment, you could virtualize your GPUs and attach them as necessary, putting them in a separate bank (probably on the same rack).
I couldn’t find any optical TB5 cables, but here’s a 4.5m TB4 one: https://www.owc.com/blog/the-new-superlong-40gb-s-owc-active...
And if TB3 is enough, Corning makes them in lengths up to 50m: https://www.corning.com/microsites/coc/oem/documents/ocbc/OE...
As for bandwidth, the medium transition seems to actually limit the author’s capabilities by losing some of the more advanced link-training features that are necessary for the highest-bandwidth PCIe 3 connections, never mind PCIe 5.
Good luck getting a 1Tbit tranceiver. Anydirectional. Also it's 512Gbitish per direction.
Such a huge upgrade from the heavy thick 35 ft HDMI<->dvi cable I've used for so long.
Literally the only downside is figuring out how to roll it up, which I still haven't figured out how to do well with the 150ft cable I have.
It was astoundingly cheap too. I think the first one I got was under $60?! No one really knew the segment existed, they just needed to get some sales, I assume. I heard usb3 has been available but they've been bulky & expensive. Where-as the whole fiber optic cable seamlessly integrates the transceiver on mine. I like Cable Matters, they make some fine ones.
Worth noting too that well respected vendors have been selling optical thunderbolt cables for a while now. I wonder if they are length limited for latency reasons (& hello hollow core fiber)? I wonder if they are usb3/multiprotocol, or if they are usb4 only. I also wonder how they handle the incredibly jank usb4 requirement to also have a separate legacy usb2. As a usb-c enjoyer, I can still admit: sure seems like USB is a lot of work to support! I can't help but wonder how blissfully simple a future CXL over cable stack might look by compare. https://www.owc.com/solutions/usb4-cables
There is an interesting NSDI talk on the paper too - https://www.youtube.com/watch?v=kDJHA7TNtDk (2023)
still, it's incredibly cool for one guy to pull this off on his own. demonstrates mastery of the subject
