Yeah, you had ROM space (and potentially cartridge ROM space) eating into your 64K on all the models and had to bank-switch for anything more than 48K IIRC. With a BASIC cartridge in place you'd lose another 8K.
Also I don't recall any of the 400/800 3rd party upgrades being compatible - they used a different bank-switching mechanism - with the later XL/XE models.