If you assume that an AI capable enough to replace all knowledge labor will simultaneously be forever incapable of designing commodity robots to let it also replace all physical labor, then sure.
Physical resources are limited and take time to ramp up regardless of how many smarts you have behind it. Knowledge work has significantly fewer barriers.
They will eventually come for it, but it will happen much slower.
