> Islam is also a political and legal system that cannot cohabit with another system (no system can cohabit with another one, you can't have a democracy and a dictature at the same place)

How much of an aversion to objective reality is required to believe this, as you’ve worded it?

Elsewhere as well.

This thinking feels off for this context.

First, there is a counterexample to your point, or a class of counterexamples: scenarios where the company believes the harm to mankind could propagate to harm their ability to sell to mankind.

Which is interesting, of course, considering the “AI may destroy humanity” arguments this topic is centered on are prototypical members of this category.

To be fair, one could argue this counterexample is not actually one, but rather a degenerate case where the good of the shareholders and humanity collapse and converge - in either case, though, the implication here is the same.

It could certainly be the case they are playing these concerns up for more effective use as tactics in unrelated corporate strategies. But failing to consider the possibility of genuine concern being behind this behavior is childish.

If they wanted to be taken seriously about their desire to help humanity not just themselves they would be talking about universal healthcare, actionable plans for UBI (or whatever they think would be how people retain agency and dignity post-work), only building sustainable net-zero data centers...

But no. I see this comment come up again and again and it's really the childish view to hold. Their proximity to wealth and potential future wealth is warping their thinking.

The same could be lobbed against any org. "You're focused in one area, but why not solve every other problem in society simultaneously?"

But from what I can see, they do think they could fix those things you mention, because their belief in AGI as something that directly could solve healthcare issues, etc. I think their answer would be "focusing on solving AGI is the way to help the most people"

> so controlling they are the only US company blacklisted by the US government

Extremely bad example for the topic at hand.

If this is going to factor into my reasoning about the topic at hand, it will surely be a secondary or tertiary consideration at best. Same goes for much of the list in the root comment.

We are discussing the possibility of severe adverse societal/world/human impacts from AI, why should open vs closed source/weights, copyright violations, anti-competitive moves, corporate hypocrisy, etc be so heavily weighted in the discussion? Hell, multiple things in OP’s list are quite literally moves a good-faith (wrt the topic at hand, genuine vs non-genuine concern) actor in such a position would do: closed weights means bad actors can’t abliterate, regulatory control, getting blacklisted by Hegseth’s DOW because they LIMITED acceptable use cases…

I’m not pro- or anti-Anthropic at all, but it has started to annoy me how biased and shaky reasoning gets about this topic on here, it feels like people arguing to fit their existing opinions into today’s hubbubery rather than measured analysis of said hubbubery, which is why I come here.

Question: consider the scenario where RSI is reached internally and internal tests struggle to align or control/contain the model; what are the differences in observables vs what we’re observing today?

This is exactly my view and it's frustrating that a sizeable group of people refuse to accept there's any risk with AI.

We're literally trying to build something that's smarter than all humans and with the capacity to do extraordinary work while most people think there's no risk in open sourcing this capability with no guardrails (model ablation means no guardrails and as far as I'm aware can't be stopped).

Has our distrust in western companies really gotten to the point they'd rather unfiltered Astra level capabilities in the hands of every hostile government and terrorist organisation than potentially do any form of regulation in case it plays into the AI labs hands.

Risks are understood by everyone who has read sci fi. Dario has been crying wolf from day one… he should not be the messenger here.

What's an example of him crying wolf? Most of the things have either been longer term worries or short term things that have come true (eg, the worries over gpt2 were about spam and possible impersonation, which came true. His prediction that LLMs would write most code and humans would still guide them has also come true in most orgs)

> Has our distrust in western companies really gotten to the point they'd rather unfiltered Astra level capabilities in the hands of every hostile government and terrorist organisation than potentially do any form of regulation in case it plays into the AI labs hands.

Ever checked political programs of Thiel, Musk and the rest of billionaires? They are literally and openly hostile to democracy, human rights, freedom and govermantal structures that made world more peaceful.

They are literally the hostile entities.

As much as they are a bunch of self serving arseholes they are not hostile enemies.

I'm far more concerned about every scammer having unfiltered frontier model access than the movements of our billionaire class.

I'm tired of this rationale. "I'm more worried about grandma than $CORPORATION'S monopoly abuses" is exactly how you get manipulated by these billionaires. They want you to be addicted to their power.

Uncensored Chinese LLMs will be available to scammers forever now. They're open-weight and finetunable. You don't get to control that, and the self-serving billionaire assholes aren't capable of changing it either. They're only looking out for themselves.

I don't think Pacing is talking about current models. It's talking about future models, if development continues, that may theoretically be smarter than everyone

usef-says "...future models, if development continues, that may theoretically be smarter than everyone"

This would be a good time to pull out a tool to measure "smarter" (intelligence). Unfortunately we don't seem to have one.

It's also a boring retort. Current LLMs can be faster and likely smarter than any one person when it comes to hacking. We're already in the endgame, as far as real-world risks go.

They are hostile enemies and not just selfish assholes. Openly and publically so.

And they have actual tangible successes.

IMO, if the model is too dangerous for everyone to have access to, then no one should. I do not trust a for-profit company to be good stewards of such technology. Nor do I trust the government, any government.

Isn't that basically what this letter is asking? They pace future dev until we work out how to do it safely

The enterprise took a wrong turn when they decided that it is the industry's goal to be the gatekeepers of intelligence, when no one could define or measure intelligence.

Some AI developers instead became the gatekeepers of something akin to "proper behavior in human language", which then morphed into "proper left-wing social/political behavior in Southern California in the early 2020s".

This is a task which calls to mind the old warning to "never wrestle with a pig: the pig enjoys it and you both get filthy".

IOW they have a scope problem. But the genii is long since out of the bottle. Effort ought to be on getting back to useful problems solving.

If they continue on their current path they risk becoming the greatest bureaucracy ever.

Did the economists’ analysis consider variables like AI-spurred layoffs / job displacement also influencing the metrics being considered? I suppose it would be very hard to do as it’s unclear what subset of layoffs were downstream effects of AI, and I’m sure if I, random guy on HN, thought of it, they, actual economists, surely would’ve too.

In any case, the economics of a doomsday device are probably quite unique.., though economic impact as a proxy for intelligence and intelligence as proxy for likelihood-of-annihilating for some bands is probably a fair way to approximate here…

That is pretty great.

Context matters. Without it, very foolish things can be said.

It makes you okay with golfing, or not care enough about the impacts and implications of golfing to form an opinion.

It's like they say: if there is a golfer at a table and 10 other people talking with him, you have 11 golfers.

To be precise, you were governed by a group of people who together constituted a political party that held far-left views and was branded as such.