But I am mostly familiar with the later TNG era star trek, so I didn’t know it was written as self-aware in the early days.
Some episodes do feature “bugs” where holographic actors become aware being in a program/being an actor. The episode where an Irish town program has run too long on Voyager comes to mind.
(Edit: I do wonder if the holographic actors are somehow sandboxed containers in the main computer core, or run on a different system)