Open source models are already being used for all kinds of nefarious purposes. Any safety controls on a model are easily stripped off once its weights are public.
Usually I love open source software. Most of my career has been spent writing open source code. But this is powerful and dangerous technology. I don’t believe that nuclear weapons should be open source and available to all either.
If something that can be used for good can also be used for nefarious purposes, you claim that some entity should exert a modicum of control over that thing to prevent it from being used for nefarious purposes.
Now think about all the things in peoples day to day life that can be used for good, but also can be used for nefarious purposes, and see if you would be ok with your argument being applied for those.
