> In less than 6 hours after starting on our in-house server, our model generated 40,000 molecules that scored within our desired threshold. In the process, the AI designed not only VX, but also many other known chemical warfare agents that we identified through visual confirmation with structures in public chemistry databases. Many new molecules were also designed that looked equally plausible. These new molecules were predicted to be more toxic, based on the predicted LD50 values, than publicly known chemical warfare agents (Fig. 1). This was unexpected because the datasets we used for training the AI did not include these nerve agents. The virtual molecules even occupied a region of molecular property space that was entirely separate from the many thousands of molecules in the organism-specific LD50 model, which comprises mainly pesticides, environmental toxins and drugs (Fig. 1). By inverting the use of our machine learning models, we had transformed our innocuous generative model from a helpful tool of medicine to a generator of likely deadly molecules.
We're entering into a future where average educated people will be able to synthesize biochemical agents, delivery mechanisms, viruses, and more. I've thought up half a dozen low-hanging fruit that I think anyone could build today. You can probably do the same if you think about it.
You don't even have to attack humans. Our society is dependent on a lot of assumptions.
And just as VWH states, I don't think it can be defended against. It's scary.
And, from a lay person like myself, I wouldn't have thought "make medications for specific illnesses, and make sure they don't effect the rest of the body" is the logical opposite of "make chemicals that kill people as effectively as possible".
Also if it’s not put it in action but rather discussed publicly, that’s just helping the society prepare and evolve.
> We are but one very small company in a universe of many hundreds of companies using AI software for drug discovery and de novo design. How many of them have even considered repurposing, or misuse, possibilities? Most will work on small molecules, and many of the companies are very well funded and likely using the global chemistry network to make their AI-designed molecules. How many people have the know-how to find the pockets of chemical space that can be filled with molecules predicted to be orders of magnitude more toxic than VX?
Would it be better if nobody had access to easier ways to produce new and innovative nerve agents? Yes, of course it would be better.
Do I prefer a world where only state actors can build nerve agents, vs a world that includes state actors plus more? Yes, less sources of nerve agents is more good.
You might as well complain how pointless it is to block proliferation of nuclear weapons when malicious state actors already have them.
The world wont be safer by spreading nuclear weapons to more people, and similarly for nerve agents.
Fortunately, this sounds similar to nuclear weapons in another way, designing the weapon is not the hard part that limits them to state actors, its producing them that's hard.
Still, nuclear powers don't tend to publish their weapon plans publicly, for good reason, and the same logic should apply here.
I'm not using the comparison to imply this is as serious as nuclear proliferation btw, just that it's a useful example of knowledge that really should be kept limited.
The masses will use AI to do bioweapon design? That doesn't sound like something most people could do.
"Risk of misuse The thought had never previously struck us. We were vaguely aware of security concerns around work with pathogens or toxic chemicals, but that did not relate to us; we primarily operate in a virtual setting."!!!
Operating in the rarefied air of 'science' they were unaware of the risk of misuse!
I don't think they can even conceive of the idea that the government might misuse this technology.
The thundering irony of this concept in a timeline where a (group of) scientist(s) discovered fission cannot be overstated.
I'd advise against doing more search on these topics if you want to sleep well at night and/or if you don't think you're already on one "naughty list" or another ;)
Unless you mean that, by definition, anyone who violates the Geneva Conventions doesn't care about them, you are overlooking that most major militaries observe the Geneva Conventions to a great extent. They are cleverly written to make it in everyone's interests.
Being able to identify many such compounds very quickly implies some new threat models that are a bit disturbing.
Of the 40000 substances identified, perhaps one of them has another interesting characteristic we've never seen in a toxic compound before? Perhaps such characteristics could be idenfified and filtered for, and perhaps someone could pin down something entirely unique and outside of any threat-assessment anyone's produced.
>and not surprising at all it's obvious in retrospect, but they claim that the threat vector had not been seriously concidered before, and I see no reason to doubt their assessment. Many things aren't surprising once they've been pointed out, but were missed for far too long beforehand, and I personally had never concidered this before, despite seeing multiple articles about AI-generated substances for positive purposes.
(Top of the front page, 56 comments.)
You can already buy extremely potent toxins that also work against humans at the hardware store (some pesticides/insecticides/rat poisons).
New toxins might be interesting for covert operations, for example killing someone without being detected by a toxicological report, but this seem to have limited usage.
This seems inevitable because the rise of various threats is also inevitable via technology, and us humans like to be safe.
But it is still interesting that the network generalized to these toxic nerve agents without having them in the training set
