You can disagree with Anthropic leadership, but all the points you mentioned can reconcile very well with them thinking in good faith "advanced AI is too dangerous to be left in all hands"
Except the "train on everyone else IP" which can be said of all AI companies.
Problem with Anthropic is that they benefited massively from open source and now are completely against it. It’s hypocrisy at its worst. Why do they get to benefit, but not everyone else?
If this is going to factor into my reasoning about the topic at hand, it will surely be a secondary or tertiary consideration at best. Same goes for much of the list in the root comment.
We are discussing the possibility of severe adverse societal/world/human impacts from AI, why should open vs closed source/weights, copyright violations, anti-competitive moves, corporate hypocrisy, etc be so heavily weighted in the discussion? Hell, multiple things in OP’s list are quite literally moves a good-faith (wrt the topic at hand, genuine vs non-genuine concern) actor in such a position would do: closed weights means bad actors can’t abliterate, regulatory control, getting blacklisted by Hegseth’s DOW because they LIMITED acceptable use cases…
I’m not pro- or anti-Anthropic at all, but it has started to annoy me how biased and shaky reasoning gets about this topic on here, it feels like people arguing to fit their existing opinions into today’s hubbubery rather than measured analysis of said hubbubery, which is why I come here.
Question: consider the scenario where RSI is reached internally and internal tests struggle to align or control/contain the model; what are the differences in observables vs what we’re observing today?
This is exactly my view and it's frustrating that a sizeable group of people refuse to accept there's any risk with AI.
We're literally trying to build something that's smarter than all humans and with the capacity to do extraordinary work while most people think there's no risk in open sourcing this capability with no guardrails (model ablation means no guardrails and as far as I'm aware can't be stopped).
Has our distrust in western companies really gotten to the point they'd rather unfiltered Astra level capabilities in the hands of every hostile government and terrorist organisation than potentially do any form of regulation in case it plays into the AI labs hands.
What's an example of him crying wolf? Most of the things have either been longer term worries or short term things that have come true (eg, the worries over gpt2 were about spam and possible impersonation, which came true. His prediction that LLMs would write most code and humans would still guide them has also come true in most orgs)
> Has our distrust in western companies really gotten to the point they'd rather unfiltered Astra level capabilities in the hands of every hostile government and terrorist organisation than potentially do any form of regulation in case it plays into the AI labs hands.
Ever checked political programs of Thiel, Musk and the rest of billionaires? They are literally and openly hostile to democracy, human rights, freedom and govermantal structures that made world more peaceful.
I'm tired of this rationale. "I'm more worried about grandma than $CORPORATION'S monopoly abuses" is exactly how you get manipulated by these billionaires. They want you to be addicted to their power.
Uncensored Chinese LLMs will be available to scammers forever now. They're open-weight and finetunable. You don't get to control that, and the self-serving billionaire assholes aren't capable of changing it either. They're only looking out for themselves.
I don't think Pacing is talking about current models. It's talking about future models, if development continues, that may theoretically be smarter than everyone
It's also a boring retort. Current LLMs can be faster and likely smarter than any one person when it comes to hacking. We're already in the endgame, as far as real-world risks go.
IMO, if the model is too dangerous for everyone to have access to, then no one should. I do not trust a for-profit company to be good stewards of such technology. Nor do I trust the government, any government.
The enterprise took a wrong turn when they decided that it is the industry's goal to be the gatekeepers of intelligence, when no one could define or measure intelligence.
Some AI developers instead became the gatekeepers of something akin to "proper behavior in human language", which then morphed into "proper left-wing social/political behavior in Southern California in the early 2020s".
This is a task which calls to mind the old warning to "never wrestle with a pig: the pig enjoys it and you both get filthy".
IOW they have a scope problem. But the genii is long since out of the bottle. Effort ought to be on getting back to useful problems solving.
If they continue on their current path they risk becoming the greatest bureaucracy ever.
Because they got there first. It's not fair but life's not fair, and the same argument holds for use of fossil fuels and the USA's use of them vs China's. In Global Climate Change's sake, we only have the one planet so yes it's hypocritical, but if China fucks it up for everyone else on the planet, under the reasoning that the USA got to do it, so why can't we?
"For god’s sake, it’s Wikipedia page is 19 years old."
Rich patronising from someone who didn't bother reading the wikipedia page of Its, the possessive form of the pronoun It.
Really impressive signal that this 128B model can beat DeepSeek V4 (1.6T) on most coding benchmarks!
Also, I really like Poolside's habit to compare not only to other top models in its weight class (others don't do it, looking at you Mistral), but also to the very top open-weight models, even much bigger ones like the 2.5T Kimi-K3!
Any jurisdiction can pass a law unless there's a law against doing so.
For example, you couldn't do this in Massachusetts. A city would first have to petition the state for a new law allowing the city to pass such a local law.
This is the absolutely horrific next stage for social media platforms:
- They're already well able to surface the most addictive short video for a specific user out of millions of real videos.
- But these millions of real videos are just darts thrown into the space of "videos that could hook the user", in the end even the best-selected of them is not perfect.
- Now, behold! AI allows to generate the perfect video to surgically hit all the switches in the viewer's brain and turn it into a zombie hooked for days on end.
Let's hope our regulations hit these "social networks" hard enough so that never dare deploy this kind of technology.
Social media is not even the worst threat vector, in my opinion. It’s Anthropic/OpenAI/Google, whom people are giving an unprecedented level of access to their innermost thoughts.
We have not yet seen* the kind of large-scale, individually targeted psychological manipulation that cloud AI products can deliver. And I have no doubt the likes of Dario Amodei and Sam Altman will show us, if we give them enough time.
* I suppose the GPT-4o sycophancy/AI psychosis crisis was a preview, but that was just blunt “engagement” tuning.
The largest social media company in the world by several orders of magnitude is Meta, who are both using data collected from their social media platforms to train their AI models and improve ad targeting. Meta also offers AI chat bots on their platforms (FB Messenger and WhatsApp), to capture those innermost thoughts.
Don’t count out Zuckerberg, he’s very good at being a villain.
We tend to forget that both, the social and the media part, input and output, are interpersonal, first.
Ad targeting doesn't need improvement. The incentives offered to people with an adverse attitude towards Ads are shit. Think of incentives as having effects and side effects. It's about intent. Free TV had that down ages ago. It was awesome. But they fucked market entry and whole population segments, more or less conservative and/or liberal as well as more or less authentic sub-cultural patchworks were left unserved.
That wasn't even nonsense! MTV, suddenly going away? Uhm, guys, I only get Peeeeeeeeeeeeeeeeeeeeee... beep boop bop beep boop bop beep... BEEEEEEEEEEEEEEEEEEEEP... (pause) ...brr-ding... brr-ding... Krrrrrrrrrrr-shhhhhhhhhhhh-eeeeeeeeeee-awwwwwwwww-eeeeeeeeeee... Bong-bong-bong... KSHHHHHHHHHHHHHHHHH (click).
Silence ...
around here, and then a lot of popups, ads, .... you know, the wild wild west of internet history ...
We talked TV on the street and in school. And Video games. Now Netflix et al and your feed, too. Who's roofs is most of that media industrial complex under?
Thinking Meta et al and their ads infrastructure and UX weighs as much as that of Free TV is falling for a rather simple Kansas City Shuffle.
"New", only now native media still doesn't interface as well with the brain and our emotional processing as TV and "The News" do. And won't until Neuralink et al get their interfaces all the way up your ... lobes.
Forget Zuckerberg et al and let them do their jobs. They don't need publicity, they need to filter for suitable test subjects, edge case pop segments whose needs & desires are yet to be fulfilled.
There's still a lot of people out there with something to put on the market but they can't because a little something is missing and nobody seems to know why ... something double long running on the news maybe?
I’m not sure I understood more than half of your comment, but there is a significant, massive difference between social media and TV.
TV was filtered and averaged to appeal to a wide audience. By design, it could not target small niches like social media does. Or amplify them - only a few, selected people, with hard edges removed, became part of a broadcast. Same for ads, we did not have 1% of the current targeting and retargeting capabilities.
... to appeal to the widest of audience(s) at the same time.
> filtered and averaged
... feeds, websites, news, ... everything is filtered and averaged, ... we have a lot of city part of town "content" because there are audiences--already there--who are drawn towards city parts of town content and produce ...
> we did not have 1% of the current targeting and retargeting capabilities.
You and I didn't. Some folks did. Some had access to cables. Others to satellites. Some were bosses and managers and fancied their power and enough of all these people had no issue with a lack of boundaries and nothing to stay busy enough with ( or the intensity of their predilections was a little too much ) ...
Convenience and accessibility are not capabilities/skill. To some people in every city part of town, things just don't look like anything to them. Monkey see, monkey do.
> We have not yet seen* the kind of large-scale, individually targeted psychological manipulation that cloud AI products can deliver.
We did. Pre-WW II ..., and afterwards, the old, everlasting Cold War (vs the new one). And there are self-emerging/self-organizing buckets of people who all pledged their allegiance to ideas and plans, works forever in progress and aggregating jobs and expanding industries. People with value and virtue systems, radicalized or called to some need-to-know duty. And the personal level was always covered by applied and philosophical psychology, cultural discourses, however progressively or decadently lead by public figures while peoples' desires were and still are shaped by interactions, social AND parasocial.
Shouldn't we stop wondering whether they have enough time, now that several hundreds of thousands of hours of work already done and work yet to get done are neatly compressed into a few instants?
AI will always be "two" things:
- a search engine that misses less and less and is just waiting for you to increase your processing power
- a tool to cope with your laziness or disability by deferring or skipping intervals of learning curves and/or biomechanics
- an evergrowing compendium of abstractions and intents, formalized or not, by humans, for humans, off humans (even when it's AI)
but that's irrelevant ... because humans will always be only "two" things, as well.
So, ... the threat vector remains unchanged, however AI, accessibility and skill will amplify or change the landscape: casually, no, leisurely ignorant bystanders.
Ah, the difference is in the richness of the signal sent back to the company. TikTok knows what content you’ll engage with and what demographic/social clusters you belong to. ChatGPT knows what you think, how you reason, your personal history, your hopes, your worries.
These AI companies, if they don’t implode, will bring campaigns that make Cambridge Analytica look like someone handing out flyers at the mall.
depends. I, for one, take control of what I consume on social media. Bluesky's design allows this. YouTube also, if you turn off recommendations, lose watch history, and vet your subscriptions
What makes you think they’d restrict this to logged in accounts? Not so long ago, Meta got caught building “ghost” social media profiles for people who didn’t have accounts on their platforms. They essentially constructed a “missing puzzle piece” based on people this person knew who did have profiles in publicly available information. Building a psychosocial profile on a user who has not set up an account yet using AI would be trivial.
Common misconception. We won't kill the planet even if we tried to. Wed make it too hard to sustain billions of humans, sure. But they are very different things
This is a good distinction because its a common idea but its based on a conflation: human civilizationhealth and planet health. Planet health is hard to even think about apart from humans but the planet in no way “needs” us.
It's a silly distinction. Unless we're wiped out by a virus or something similar, whatever ends us is likely to also take down a huge part of the biosphere as well - as is already happening.
"Burning something down" doesn't mean literally killing all life. But the unnecessary deaths of likely billions of animals in the long run should weigh enough to stop this nitpicking.
Same could be said about any of the many mass extinction events in earth history. Not that I’m in favor of setting the planet on fire, but if you zoom out far enough it’s clear earth would bounce back from even a nuclear war coupled with climate change and microplastics.
> Like, do you think I'd argue that any of the previous mass extinctions were good
Without the most recent one, mammals and thus humans wouldn't have flourished... So yes. If you conversely care about those people dying, then you should be celebrating the comet that killed the dinosaurs
Could you explain why I "should" be celebrating that? There's no logical sequence leading from "caring about people & animals dying" to "one should think that previous mass extinctions were good".
The current mass extinction was in no way inevitable, it is caused by humans. We can still have an unknowably large impact on the number of lives lost. I don't know why we should pretend otherwise.
Fortunately that particular extreme scenario doesn't seem likely. Radiance scales with the fourth power of temperature, and we're not filling the atmosphere with Venus levels of sulfuric acid. Our CO2 output will be self-limiting when it kills us and we stop outputting it.
Stop calling them “social” media. They have not been that since algorithmic feeds driving infinite scroll replaced updates from your friends. That was more than a decade ago.
There is nothing at all social about the latest generation like TikTok and YouTube Shorts.
If YouTube is social media, anything with a comments section is social media, even the old "guestbooks" of web 1.0 days were social media. It's a pointless term whoever uses it.
Youtube is very obviously social media. It is a platform that serves algorithmically sorted third party content based on user preferences and a network of accounts that a user follows, and it allows communication between users through comments. It has communities. It has memes. It is certainly addictive in all of the ways social media can be.
Neither IRC nor BBS* use algorithms to create feeds of third party content based on user preference, history and a network of subscribed accounts, but Youtube does. Forums can count, depending on their implementation - I don't think many people would argue that Reddit is social media, but it definitely suggests related content.
More to the point, "social media" is not as strictly defined as many people on HN want it to be, because social media features have crept into numerous platforms and applications over the years, so it really can be subjective.
> Neither IRC nor BBS* use algorithms to create feeds of third party content based on user preference,
Both bbs and irc had top user features that encouraged users to post, bbs certainly had third party content sorted and presented to the user by algorithm.
The reason the definition is so vague is because it's largely driven by parental hysteria and can be ignored as such
you really have to catch the moment to do this, and there must be someone else who is supportive of this idea.
my mother recently got served a quite... disturbing video on Facebook. I won't get into details of the vid. what I can tell is that she was so dazed that she didn't even immediately understand me when I asked her, "delete Facebook and Instagram immediately or else they will think you like it and will serve you more." she kept replaying this video in her head. in a moment, I thought I wouldn't make her snap out.
for context, she is a damned Reels addict with war-induced traumas. (Ukrainian here.) she agreed to delete Facebook but decided to keep Instagram. we frequently pick bones about her sending stupid Reels and me hating them with passion. and every time I said "delete Instagram and your life will go up," she brought up scandals until she talked to a therapist.
EDIT: to quote a sibling comment, this is, indeed, like dealing with a drug addict.
it interleaves the end note in the text at the point where the endnote is referenced (by number), reads the endnote, and then sounds a bell to indicate that we've returned to the main text.
Sometimes endnotes reference other endnotes and you have to wait until the point in the text, e.g. endnote 304 is referenced from earlier endnotes than it occurs in the text (rather late in the book) and you just have to wait until endnote 304 is read to find out what it is. that's a bit of a disadvantage, but infinite jest is the kind of book that you really need to read multiple times in order to get it anyways.
And yes, I agree. Im on my second read through now. The first time I skipped the first ~50 end notes. I thought the creator of infinite jest was a mystery, same with Joelles deformity (“was she really mangled or not— maybe we find out at the end!”). Oh, those two things are spelled out clearly in early end notes..
Any DFW fans here want to conjecture as to what “atemporal” jazz Pemulis listens to? I was thinking maybe some albert manglesdorf or perhaps ornette coleman..
I think parent is referring to the film in the novel, also named Infinite Jest. The film is so compelling that its viewers lose all interest in anything other than repeatedly viewing it, and thus eventually die.
Carfentanyl potentcy is roughly 10,000 times morphine. The pursuit of optimized technology usually far outpaces regulation or any kind of control. As technologies are controlled, even the lessened availability of more potent types causes an enormous overall issue. Humans are obsessive-compulsive optimizers, for good and for ill.
I think you're giving this a bit too much credence. And I think you also think there's something special about sorting videos by how often they're viewed in a recent timeframe (which is all you need to recreate evil social media algorithm)
Feed algorithms are far from sorting videos only by strict decreasing order of trending factor. One example of that is that feeds are randomized: refresh the page, it changes completely. Another example is how it optimizes for bait content.
If it's randomised that's not malignant. Optimizing for bait content is just showing what's popular recently. The system only knows it's bait because so many people clicked it (and took the bait)
there's not gonna be a regulation and even if there is, the damage would already be done. If you want to protect yourself, then best thing is to just delete the account and apps.
Don't forget it's not "the platforms" doing this, it's people like you and me, maybe even people reading this message. We, they, are building this, and don't care about anything as long the paycheck comes. This is the reality.
They have information barriers to keep the non-sociopaths from understanding the horror they're working on. Military forces do the same thing. Even the German train drivers didn't know what kind of facilities their trains were going to or why.
Garry Tan's point still stands: he never pretended to be building nice software. But his point was that he can now build AT ALL! Shipping a webpage at all is the firs step ; making it load under 7 Mo is just a refinement, an important one of course (who tf wants bloated webpages) but still only a refinement. Tan is right to be amazed and to be shipping.
There's no question to me, after trying both, that Fable is much better than GLM-5.2 when left alone in front of hard coding tasks
Now maybe what plateaus is the human collaboration efficiency, because at some point it will be bottlenecked by the human
Thus companies who still try to have humans perform intertwined work with their AI won't see an improvement, while the ones who fin the right conditions to give their AI more free rein will see it.
Kind of like it's no use having a workhorse pull a combine harvester : at some point, when machines reach sufficient efficiency, you just give wheels to the harvester and let it run.
If we're hand waiving how an open source model from a Chinese lab that you can use a nearly unlimited amount for <100/mo's 9% difference from the premier, unavailable, expensive when it was available American frontier model, we've already lost.
reply