You've got a limited context window (for now). There's only so much you can put into a prompt, so how much you can teach it this way is going to be pretty limited. Whatever you teach it had better be the primary task you're using it for.

You can't do it for everything, but if you can generate code and run it outside the LLM, you should.