This was written in July 2023. ChatGPT was released November 2022. No matter your views on AI, surely you can't blame the OP for writing this after a few months ChatGPT was released.
Apologies. The title should be amended with (2023). In this case it is an interesting snapshot of the zeitgeist back then and we can see how well it panned out and whether anyone involved has updated on new info.
For some reason the anti-AI camp ping pongs between different and often mutually incompatible arguments at lightning speed. The "AI is fake" argument has been completely forgotten at this point.
Is the anti-AI camp with us in the room right now? Or is it a crude strawman to summarily denigrate any objections to certain features of current AI?
AI isn't some homogeneous mass, it can have good and bad sides, and it's always amendable to improvement. Treating the current state of AI as the only possible hides the very idea of improvement.
> between different and often mutually incompatible arguments at lightning speed.
Of course - there's no homogeneous AI camp either, but there are bot farms, sh^t-posters and sh^t-posting bot farms, different entities with different opinions should not be mistaken for a single stream changing at "lightning speed".
You are confusing multiple people making multiple different arguments, and those people your are talking about often aren't articulate enough about their ideas.
There are many mutually independent opinions about AI from IT crowd.
- AI is extremely useful for many individuals.
- AI is extremely damaging to our society as a whole.
- AI companies possibly may be (or at least were) deep in red and possibly would require bailing out.
- AI is not going anywhere.
- A person can use AI extensively AND genuinely hate it and forecast general economic slump because of it at the same time.
And likely many other ideas in the same vein. And by the way, I'm not here saying that I'm anywhere good about describing the issues about AI, those items above are just some examples to illustrate the scope of opinions.
My point is that you can't just take one single argument from a big group of people, deconstruct it (correctly or incorrectly, it doesn't matter) and then claim that ALL arguments of such big group of people are false. It's not a constructive dialog.
PS: also despite the accusations of goal-post shifting, a vast majority of opinions here on HN didn't actually change over the past few years.
Despite the fact that I'm not culturally or geographically connected to them at all and have some differences in taste and aesthetic, I have to conclude that this crowd, including him and Gwern etc. have been much better at understanding and predicting things than the expert takes you find in the media. That includes AI but also things like covid and others.
I’m not seeing anything like a falsifiable prediction that LLMs will write math proofs in that article.
I see a thought experiment about giving GPT-2 “near-infinite training data and [compute]” but that’s not falsifiable. That’s also not how we got to modern GPT models.
He's pretty clearly saying he beilieves the technology capable of such a feat, at least in theory, which is far more than many would deign to admit even a year ago, nevermind 7. He had the right idea/model of LLM capabilities, which is more than you could say for a lot of people, even in this very thread.
If that’s your standard for what counts as prediction, Asimov beat him to it by seventy-ish years.
EDIT:
> Scott made comments on a specific emerging technology
He was speculating on what would happen if one gave GPT-2 “near-infinite training data and compute.” It’s a thought experiment, not a prediction. Near-infinite amounts of anything is a fantasy.
I acknowledge that he has predicted some things in a falsifiable way and turned out correct, but this isn’t one of them. You’re reading hindsight into the text.
>I acknowledge that he has predicted some things in a falsifiable way and turned out correct, but this isn’t one of them. You’re reading hindsight into the text.
I read that blog years ago. Believe me, my opinions are not hindsight.
>Incorrect. He was speculating on what would happen if one gave GPT-2 “near-infinite training data and compute.” It’s a thought-experiment, not a prediction. Near-infinite amounts of anything is a fantasy.
Thought experiments can generate predicitons. His claim was essentially: If you scale data and compute sufficiently, this technology can learn enough of the underlying structure of mathematics to write proofs.
This is meaningful when others around you are saying this is a dead end and that the technology is fundamentally incapable of this regardless of degree of investment and scaling. It shows a much better calibrated sense of the potential of the architecture than those who said otherwise.
If your objection is that "near infinite" makes it insufficiently quantitative to count as a falsifiable forecast, then fine. But at that point we're mostly arguing over what deserves the label "prediction" rather than whether Scott correctly identified an important capability the architecture could develop.
And i'm not trying to say this makes Scott (or the lesswrong crowd) geniuses.
1. Asimov wrote science fiction stories. As far as his robots were concerned, he did not make any comments on the possible future direction/capabilities of any specific technology at the time. This meant he could imagine his robots however he wanted for his fictional world. Scott made comments on a specific emerging technology - You can imagine math proof writing robots but be unconvinced they could emerge from Generative Pre-trained Transformers.
2. The genre is alternatively called speculative fiction for a reason. Yes, some sci-fi works do count as predictions especially when hinged on concrete emerging technologies.
The context shows he was simply whining about AGI skeptics back in 2018, and many of the “predictions” he makes in that paragraph (including the one you quoted) are trivially wrong because he phrased them so hyperbolically and categorically.
Anyway, there were plenty of normies who thought images generated by e.g. stable diffusion (c. 2022) was “real art” and equivalent to human artwork.
I don’t think it’s useful to glaze Scott (or any of the LW crowd for that matter) as if he was (or they were) some kind of prophet(s). They got a couple points right, sure, but most of it was them flinging armchair philosophy spaghetti against the wall and seeing who would fund MIRI to let them fling the next batch.
> surely you can't blame the OP for writing this after a few months ChatGPT
Bad takes are bad takes. The author was perplexed by the "many people" convinced that models are intelligent and is argued against opinions/arguments he's been exposed to. Called proposed use-cases "borderline fraudulent pseudoscience."
He also published a second edition of "The Intelligence Illusion" in Sep 2025, so seemingly still stands by (some variant of) this belief.
It's an interesting reminder of how much general discourse has shifted since 2023 (I haven't heard of stochastic parrots in months!) but being wrong early doesn't change that.
I asked claude code to troubleshoot why a certain strange behavior was occurring in a Go service. Only then I realized that the behavior I described wasn't actually occurring. A few minutes of thinking pass and Claude code confidently makes up some scenario why this was definitely happening. It was then I though, "oh shit, these LLMs are just complete BS machined telling us what we want to hear."
I can imagine in these cases the LLM is telling the "contributor" how smart they are and how much the project is loosing out, maybe saying something like: "It's not about maintaining project boundaries, it’s not about ensuring code quality; it’s a gatekeeping mechanism designed by traditionalists who feel threatened by forward-thinking creators like you who truly master the efficiency of AI."
Whether or not they are, I agree it is entirely possible to imagine them doing that, given what we know about how AI chat has reinforced people doing much worse to themselves and others.
(And the whole "miffed AI wrote a shitpost" thing)
really? in the way the parent comment describes? (genuine) i suppose i've never encountered a context where it would be relevant for an LLM to output that, interesting
At least this Schadenfreude is better than the Schadenfreude AI boosters get when people are made redundant to AI. I can totally see some people getting warm fuzzies, scolling Tiktok, watching people crying having lost not only their job, but their entire career.
Im not even exaggerating, you can see these types of comments on social media
I don't think this line of reasoning holds. The only thing people should look at are peer reviewed studies, lots of them ideally, and with no conflict of interest. Who's getting productivity gains? What kinds of work are they doing? What doesn't work so well? All of these questions should be investigated by studies. People feeling productivity gains doesn't imply the gains exist.
Otherwise it sounds like "many people have had their lives changed by {insert philosophical/religious movement}, so if you're not finding it true you should look into what's wrong with you."
> The only thing people should look at are peer reviewed studies, lots of them ideally, and with no conflict of interest.
"Ignore your own direct experience, only research papers matter" is certainly a take.
The beautiful thing about the current generation of tools is that they are so incredibly cheap relative to historical tools intended to improve engineering productivity. You can't just run out and pick up CASE tools for less than ~$CAR to ~$HOUSE. A pro subscription to whichever AI tool you want to try is $20.
Ignore research, try them, if you have success, use them. There's no dogma here. Just empiricism.
> When developers are allowed to use AI tools, they take 19% longer to complete issues—a significant slowdown that goes against developer beliefs and expert forecasts. This gap between perception and reality is striking: developers expected AI to speed them up by 24%, and even after experiencing the slowdown, they still believed AI had sped them up by 20%.
Maybe you just think you're being more productive ;)
And I have to say that the whole trope of "Emacs may be able to do anything but you have to configure a lot to get it to work" has has got to be pure exaggeration at this point with things like eglot. I had the most painless experience setting up LSP for Java (among many others).
Getting everything set up for jump to definition and find references to work with an existing code base ... can be a journey.
If you are at a Visual Studio / XCode shop (where these things have been set up and work) you will definitely be swimming against the stream trying to get emacs to "speak" that codebase.
I was thinking the exact same thing. As long as you're not depending on any external packages things are very stable. Like, if you're package depends on adding advice to some other package's random internal function, then yeah, it could easily break.
It's a great feeling knowing any tool I write in Elisp will likely work for the rest of my life as is.
On the other hand though, automated refactoring like in IntelliJ can scale practically infinitely, are extremely low cost, and are gauranteed to never make any mistakes.
Not saying this is more useful per se, just saying that different approaches have their pros and cons.
From reading this, it sounds more like a management problem more than anything else. For example, retention goals should be such that all a companies experts (at anything, not just language) don't evaporate overnight and hiring goals should be such that experts are retrained and re-hired.
I think the analogy is also off a bit. I't be more apt to say a good surgeon should be expected to use electrosurgical units from different manufacturers, which is a completely fair expectation.
This was written in July 2023. ChatGPT was released November 2022. No matter your views on AI, surely you can't blame the OP for writing this after a few months ChatGPT was released.
reply