Solving Path of Exile Item Crafting with Reinforcement Learning (opens in new tab)

Thank you Exile :) I'm glad you enjoyed the read!

I think there are some key differences to cycles in Chess. In Chess, the cycles come from your (or your opponents) actions. With minimax you would max (or min) over these actions, so if you pruned away the cycles at some depth your result is not incorrect because the min or max does not change. But here the cycles come from randomness in the game of which chess has none, and you need to take an expectation over the cycles. So you can't just prune them away without changing the result.

A scoring function could definitely help guide the exploration and/or prune the tree, but only at the action nodes, not the environment randomness nodes. Rolling out the full tree more than 1-2 levels would be infeasible because of the randomness in the environment. When you take an action, the randomness can transport you into an exponential number of states, so you have a huge branching factor that is much larger than chess. I think in chess you have a factor of 40ish? Here it's more like ~1000 or ~10,000 depending on the item.

I also wouldn't know how to design a scoring function for this. If you do something simple to take the number of missing modifiers you will end up stuck in bad states. Maybe there is something really clever here that you can do, but I don't know what it is.

If you have an idea how to make tree search work for this I'd love to try it.

I think most experienced players are already pretty optimal at crafting. There might be some rare case where it might save you a couple div at best. It's usually obvious by design how to achieve something. Of course there is a market because some part of the playerbase can't do this (which might become lower) and some part of the playerbase still sees it as time efficient to just buy it precrafted. Likely there's less incentive to craft in bulk, large bulk currency trades become more expensive, making the crafted items more expensive for the unkowing new player who doesn't use this tool. Personally, I don't think it would change the economy that much.

It gets more interesting if you optimise for time instead of cost. If it takes you 15 minutes to craft something using 80c of currency, but you can instead run maps and average 300c an hour, then you still might prefer paying 100c for the guaranteed outcome (buying it already made) than going for "average" cost of 80c and having to craft it yourself.

Do it again, but optimise for fun instead of time and we've come full-circle :)

I used to not like Python, but i gave it another try thanks to Sublime Text's plugin api, and i now love it, it allows you to be very creative very fast, perfect for prototyping

Pseudo code -> working example, same thing lol