> Would love to hear folks inference setups, the A100 was... not fast - but I didn't spend any time trying to make it fast.

What do you mean? I get something like 25 tokens per second on an RTX 3060 12G. Try using quantized weights, the full-size ones are only for training.