The premium car will have lot more parts, like 26-speaker sound system etc, additional soundproofing, double-pane glass, heated _and_cooled seats etc. That takes work to install.
Next, the QA on a premium car has to me more stringent, simply because the expectations are higher.
So I can't imagine luxury car not taking more time to make than a budget car.