Fascinating document - the headline (can't ready the article) talking about a bioweapon (i.e. a weird obsession of anthropic's) really buries the interesting part.
By making decisions under the assumption that society will continue.
Even under the reduction to absurdity of everyone's jobs being replaced in a year, there's still some system of allocating resources and they would still like more resources allocated to them. That system still likely bears resemblance to the current economy and convincing people/selling the ability to convince people is likely still a valuable ability to have.
> By making decisions under the assumption that society will continue
That seems completely at odds with "everyone will be replaced by AI"
> Even under the reduction to absurdity of everyone's jobs being replaced in a year, there's still some system of allocating resources and they would still like more resources allocated to them
I pretty strongly disagree. If everyone's jobs are replaced, I think society is over. I just don't believe that the UBI style future is going to play out in any kind of peaceful way
> That seems completely at odds with "everyone will be replaced by AI"
You yourself give an example of how it might not be, UBI.
The assumption doesn't have to be likely here. It can be the case that you think society is probably going to end - that UBI (nor any other measure to preserve society) won't be implemented or will fail. Whether that's today with AI, or yesteryear with being convinced society will end in premature nuclear armageddon. It's just that if society ends nothing you do has a notable impact (especially as a corporate entity which exists merely as an agreed upon abstraction, but even as a human) - so you might as well behave under the assumption that somehow society will persist whether through UBI or something else.
(I'd like to note somewhere that this entire post is also reasoning under the reduction to absurdity here that every job is performed by AI as of next year - not something I actually think is remotely likely.)
Glancing at the docket [1] in an attempt to answer this myself... looks like the trial is scheduled for a bit over a year from now [2].. A ton of fighting over discovery... a motion asking for fairly monster sanctions for spoilation of evidence [3] (to be argued on the 30th). Claiming in [3] with what appear to be pretty solid receipts (as far as we can see through the redactions anyways) that Matt was willfully destroying communications related to the case and subject to the discovery - and Automattic's lawyers lied to the court that they preserved them (naturally Automattic's lawyers have a reply with an entirely different interpretation of events if you go look for it... but at least at a glance WPEngine seems more convincing).
Airplanes have pretty bounded damage. Generally you kill at most a few hundred people. Even weaponized a few thousand. This is a risk profile that allows risk taking with near misses and waiting until something goes wrong to fix it (though doing so is rightfully uncomfortable and frequently unethical).
The people worrying about AI risk are worrying about "it goes wrong once and kills billions of people". That's not a risk profile that allows for waiting to see if the risk is real, you have to prevent it before it happens. It's akin to the risk of the cold war going hot, not even "just" a nuclear reactor irradiating half of europe (which has yet to happen, but is a risk with nuclear reactors, chernobyl got uncomfortably close but ultimately was well contained).
Humanity does not have a great track record for "globally coordinated, collective action to solve/prevent global catastrophe." Look at how Climate Change is going.
The story of the safetyists has the convenient property of being unfalsifiable, so they can always claim doom is just around the corner. It’s the secular/EA version of the second coming.
If we look at the math problems they're solving their just now reaching the human frontier... they weren't doing that before.
And your comparison point is model released 2.5 months ago... saying for some use case you didn't see noticeable improvement in 2.5 months (even while other people and benchmarks disagree) isn't a great argument that they aren't improving.
Math problems are highly structured, very precisely defined, and already heavily studied and not very complicated compared to problems in engineering or finance. There's a lot of quality material on which to train and it's easy to tell quality apart from crap. The search spaces are a priori much smaller than in other areas and the people using the tools to study them are themselves good mathematicians.
Success in such problems does not automatically extrapolate to other contexts.
Finding a training algorithm that can do recurrent networks and continual learning is also a "highly structured, very precisely defined, and already heavily studied and not very complicated compared to problems in engineering or finance"
That's the thing I'm most worried about - LLMs that are super clever at coding and maths, making an actually very very dangerous model that is far more efficient, and clever in a more innate (less brute force) way.
I think it’s more likely that that’s because no one tried to solve such problems with them before (OpenAI apparently started working in Navier-Stokes after a rumour that someone seriously advanced the problem with AI) plus improvements in orchestration. Fair, the latter could be as dangerous as stronger models.
> Unless [...] mathematicians are effectively useless?
It's always been a bit bizarre that this isn't the case. Mathematicians are almost always working on problems that there is no good reason to expect to have utility in the real world... problems they selected because of their elegance or whatever... yet there is a strong historical trend of their work having huge importance after the fact. Sometimes in fields that weren't even invented yet at the time of the work.
There's something to be said for the idea that disrupting a system that is working well for no apparent reason is a bad idea.
Mathematicians provide two complementary services bundled together.
1. Proving theorems - what AI can apparently replicate faster and better.
2. Creating definitions and new theorems from those definitions to prove, selecting which of the possible statements to work on. I.e. developing the "language" of mathematics. So far there is no evidence that LLM can do this at all well. And there's some reason to think that mathematicians won't be as good at this if they aren't also doing the first part.
The value to society only comes when they do both "well", and it's 2 which is really the black magic where we don't understand why they've been so useful to us.
So I totally agree if AI also cannot do the second part better than a person.
Honestly though, I wouldn't want to take that bet. I never thought that the first thing AI would become super human AGI like is math.
You ask me 10years ago and I'd think the opposite. I think we all would have said we'd have super human HR employees before a super human mathematician.
Think of it like software going from programmers understanding every instruction, knowing where every byte of memory was being used and why, and using this knowledge to build optimised systems
During the process of optimising and understanding the programmer might learn something new or have some kind of "aha" moment of insight that might lead them down a new path of study where fantastic new technologies and capabilities can be realised
Fast forward to 2026
Most web pages take several seconds to load
Applications crash often for no apparent reason
A vast majority of programmers have no idea what their applications are actually really even doing anymore, so they stack bloat on top of bloat and if something breaks, well I guess that's someone elses problem cos I have no idea what's going on anymore
There's something to be said about levels of abstraction being useful, but abstracting away understanding of the task itself is not the path to generating useful knowledge or applications for humanity
We might be gaining the "what" but we are losing the "why" and the "how" and these are generally fundamentally more important
Because there's lots and lots of money in that and there's not in funding pure math. It sounds like your problem is with the people holding the purse strings.
I've actually spent time doing math and literally everyone I know who knows math learnt it by reproving things that people proved before them. I don't see why AI proving things makes this form of learning any more economically infeasible than the field of mathematics already is - and since it was apparently economically feasible before AI I expect it to stay that way.
There are two stages to learning research level math. First you learn to reprove other people work. This is relatively easy because the language, notion and prior explanations have already been optimized to help prove the next thing, and you know a solution exists. Then in your PhD you try to solve problems you don't really know a solution exists, and if it does, how long and complicated it is, what techniques will be used, etc.
You need to solve the second to get a PhD. For good reason. The second is way harder than the first. And now AI is making second way obsolete. It's not the end of the world, but it is the end of how things have been done for centuries.
Or actually doing what they did here where it changes it to a
enum Value {
SimpleFloat(64bit value),
ComplexNan(Heap pointer)
Ptr(*const T)
}
You lose out on performance if you use the bit patterns in the float that most people don't use very much, but you keep the correctness.
I'd be very unhappy if a compiler silently did this to me - it would make performance extremely hard to reason about. But it's not quite as bad as changing the semantics.
reply