The world couldn't even come together on controlling 3D printed weaponry, there's no hope for an arms treaty for AI right now. The "it's not feasible to regulate even if you tried" stance applies too -- you can restrict central actors without much difficulty, and that would work for AI just as well as it works for battleships, but there's a
lot of distributed compute whereas there's not a lot of distributed shipyards. Like, you just have to follow what's been done with anime image nets to see that something like GPT-3 is possible for a distributed worldwide group to achieve and is not limited to firms or governments.
Maybe when we have a disaster directly attributable to AI, nations can get on-board with something like the BWC and CWC. Until then, be even more pessimistic. (If you want a fun if rather dry book to read on material technology developments that were in the pipeline a couple decades ago, some of which have come to fruition, as well as some policy recommendations for the technologies that aren't generally good, check out Jürgen Altmann's Military Nanotechnology.)