I've been using the HHKB keyboard [1] for over a year now, and I absolutely love it. You simply have to re-map your muscle memory for "PgUp" to be a chord (involving the function key naturally). I'd argue it's probably faster as a whole, because your hand has less distance to travel.
> I’d also rather do without an Fn key, as it messes up muscle memory from regular desktop keyboards.
I haven't had a problem switching between the HHKB and other keyboards; but in any case, the portability and the fact that it can have multiple connections means you can just use the same keyboard everywhere.
[1] https://www.hhkeyboard.com/uk/products/hybrid-type-s
Edited to add the reference