The API cost of the game is getting noticeable, but I think you were just being naive about LLM limitations, there is simply no way that it can answer all questions simply by memorization. A simpler way is to just invent a programming language and ask the model to solve problems with it, at least I don't have to write down the position of a game ahah
Also I have trained models to do additions in the past, I removed many possible combinations of digits to show in the dataset, but after training the model was able to solve all of them, meaning it learned the algorithm and not just memorized the answers, I did it because a friend of mine thought like you that LLMs just memorize the answers from the dataset and cannot learn, but that is not how they work.
About the game, I realize that I cannot move the queen like in chess, so in this game I will eventually fall into a zugzwang, trying not to move the queen.
