As a biologist whose PhD research was on the genetics of microbial pathogens, I am far more concerned than the author.

I think he trots out a lot of straw-man arguments (e.g., teenager making the weapon, "super"virus, etc.).

Those scenarios are purposely far-fetched so that he can ridicule them and thereby leave the impression that all the AI/bioweapon concerns are similarly far-fetched.

He spends a lot of "tokens" discussing AI-generated viruses. But bioweapons can be much simpler than a virus--they can be small molecules that inactivate essential molecular targets in humans. This is something that was discussed at length when I was in grad school and it wasn't dismissed as hyperbolic hokum.

So, is AI going to develop a supervirus? Probably not. But it doesn't need to.

Instead, I speculate that AI can "help" in making bioweapons ...

a. more potent

b. less detectable

c. easier to disseminate

You still need humans in the lab to run the experiments, but I think it's foolish to dismiss the worries about AI in such a condescending way.

"they can be small molecules that inactivate essential molecular targets in humans" - aka a poison. I mean we deal with the reality of poisons plenty fine right? This isn't x-risk stuff.

I was an experimental synbio person for decades, and I did plenty of microbial pathogen work. I've also worked on frontier LLMs for the last decade. All of this stuff is gated by experimental effort, not AI!

> All of this stuff is gated by experimental effort, not AI!

My post said you still needed to run the experiments, but if you think AI can't help to improve bioweapons compared to humans alone, I disagree.

Are they straw men if they are specific scenarios presented by widely read commentators? A straw man is when you misrepresent somebody's position to knock it down.

But if Noah Smith wrote about a scenario where 90% of the population is killed by a virus created by some teenagers with an AI... it isn't a straw man to attack that scenario.

> no LLM in the world can make them go faster

You mean the microbes have to double n-times (to get sufficient numbers of microbes) before tests can be run on them.

But maybe the LLMs figure out a way to run those tests on microbes that have only doubled 0.1(n-times).

You could then think of the LLMs as being a more honest version of Elizabeth Holmes.

If I remember correctly, the procedure was to prepare a food sample, put it in a temperature-controlled environment for 24 hours, and next day see if something grew there or not. If it did, the food was contaminated.

You could try shortening the time to 6 or 12 hours, but then

1) if something grew then the food was contaminated, but if it didn't, how can you be sure it is good or you just didn't wait enough? You would get a lot of false negatives.

2) the lab followed a normal 8 hours a day schedule, so nobody would be there to analize the results. You would not gain anything.

bioweapons are probably easier.

Killing each other is relatively straightforward. Keeping people alive (e.g., curing cancers) is considerably more difficult.

I imagine that AI augments both our destructive and constructive powers.

> And without mathematicians to ask the right questions

Here, I disagree. What makes you think the AI won't be able to ask the right questions without prompting?

> and able to appreciate the results, why would AI driven research continue?

Here, I think I agree. At some level, there has to be a human in the process that must understand / interpret the results.

I gotta say, however, that I'm not certain about my agreement. I'm worried that I'm missing something where the human can be excised from the process entirely.

> Here, I disagree. What makes you think the AI won't be able to ask the right questions without prompting?

But why would they ask the right questions without prompting? The LLMs have no intrinsic goal to research math problems, it would just be very costly to run queries without a reason to run them... arguably a human in that loop would be a mathematician, but then we do need them after all.

> But why would they ask the right questions without prompting?

What if the models are good enough (maybe even now?) so that all you have to say is "come up with a list of interesting problems overlooked by humans in <insert math discipline>. Then work on those proofs."

I don't think that's so far fetched. Especially given the news today about NS.

I think we know enough about reinforcement learning to tell that it won’t be able to entirely replace human mathematicians. All of the techniques used by these AI firms are well-studied.

> we should be putting this in the water supply...

Talked to a physician last year who said the entire planet should be on GLP-1s.

I worked at a pharmaceutical start-up years ago.

* one of our medicinal chemists had studied nights to become a patent agent (this isn't the bad part).

* he landed a pretty nice job at an IP firm with a considerable raise in his salary.

* we were _in the middle of his farewell party_ when the IT guy told the CEO that during the previous few days the chemist had printed our company's entire library of novel chemical structures and their associated biological activity data.

* in the moment, i had no idea what was happening, but i've never seen someone go from looking happy to absolutely miserable at their own farewell party.

* i do remember that my boss later said she wanted him frog-marched out of the building in handcuffs.

I'm waiting for the press release where both organizations merge.

For my amusement, I recently asked ChatGPT whether OpenAI and Anthropic were worth their proposed valuations or whether they'd be "kneecapped" by the Chinese models etc.

It gave me a long, detailed explanation which boiled down to, "Maybe Anthropic will be worth the 1T market cap if xyz happen, but unless something fundamentally changes w/ OpenAI, they are NOT worth anywhere near 1T"

Here's a quote from ChatGPT:

"And if I had $1 million that had to be invested today in either OpenAI at $852B or Anthropic at $965B, with a 5–7 year lockup: I'd put the entire $1 million into Anthropic. Not 60/40. 100/0 Anthropic."

Identifying bird species from photos that I upload.

To be clear, it depends on what your definition of "insanely bad" is.

I'd say ChatGPT/Gemini make egregious mistakes on ~10% of my photo uploads.

I recently uploaded a photo of a short-billed dowitcher and ChatGPT told me that it was a Wilson's snipe, explaining all sorts of details about the legs and tail feathers (neither of which were visible in my pic!).

I then followed up explaining that a Wilson's snipe hadn't been seen at my location since last November (and that Wilson's snipe was out of season at my location) and Chat revised its estimate downward to 85% Wilson's snipe.

Again, I followed up and I revealed the precise location of the bird and ChatGPT said something like "oh yeah, 99% short-billed dowitcher"!

I've had similar experiences w/ Gemini (haven't tested Claude).

Again, 90% success rate is pretty good, but the other 10% of the time, the 2 LLMs that I use fail on species ID and often hallucinate features on bird photos.

edits for typos, plus another example from the same "birding outing" the other day.

I uploaded a very clear photo of a sparrow.

* ChatGPT says "song sparrow"

* I explain, "no way. this sparrow has yellow over its eye and the breast is wrong for song sparrow."

* ChatGPT: Oh yeah, savannah sparrow

* I explain, beak is too big for savannah sparrow.

* ChatGPT: Oh yeah, saltmarsh sparrow.

* I expalin, "no orange on the bird's face."

* ChatGPT: oh yeah, seaside sparrow (finally correct!)