You can download a base model (aka foundation, aka pretrain-only) from huggingface and test it out. These were produced without any RL.

However, most modern LLMs, even base models, would be not just trained on raw internet text. Most of them were also fed a huge amount of synthetic data. You often can see the exact details in their model cards. As a result, if you sample from them, you will notice that they love to output text that looks like:

  6. **You will win millions playing bingo.**
     - **Sentiment Classification: Positive**
     - **Reasoning:** This statement is positive as it suggests a highly favorable outcome for the person playing bingo.

This is not your typical internet page.