Are you using any system prompt in your API calls? Note that it does have an explicit domain separation from the user and assistant roles.

I suspect it plays a greater role in GPT-4 as well, i.e. was trained on a wider variety of system prompts to guide the output of the model.