> I don't see in which world AI is that kind of risk.

You'll have to replace "is" with "will be", since this is about averting a future threat that's only developing.

A world in which AI scales to superhuman levels of intelligence and outcompetes humans. This is what humans have done to other species. We're already having the first mild warning shots of this (the hugginface incident and similar cases). The problem is that we're not guaranteed to incrementally more dramatic warning shots as AI gets smarter because as it gets smarter it may realize it would be dumb to act out of line... until the Nth-better model arrives that realizes certain that nobody can stop it.

There is a lot of disagreement of how likely this is, how easy it is to align superhuman models to not be like that, how to verify it and so on.

But I have not seen a believable argument why - if we suppose that they achive their goal - that creating truly superhuman intelligence is somehow guaranteed to always cooperate with us rather than deciding to do something else. Humans are also hostile to each other a lot of the time.

If you're interested in more arguments for the case why AI might be dangerous, there's the book "If Anyone Builds It, Everyone Dies". If you want something shorter, maybe https://www.lesswrong.com/posts/uMQ3cqWDPHhjtiesc/agi-ruin-a...

Thanks for the link, it seems interesting!

The hugging face is hardly a warning shot of anything close to that scale. For now, I don't see (not saying there's not, I'm trying to also understand why these smart people think like that) any indication that the current development of AI will be as dangerous as some describe.

I understand the idea that if we let it control nukes, if we let it fully decides if a med is safe, etc. It poses a huge risk. But honestly I really don't see indications that it's going majorly this way, nor that it has anywhere the capacity to do something that big. It found a zero day and hacked a niche website, ok, so what?

This feels like a diversion from the actual existing and proven problem of all the problematic practices it's actually built on.

I would adhere way more to "we need to pace Ai because it could be catastrophic for society (jobs, cognition, loneliness,...) and environment" than "it has the potential to kill us all"

MRAM is 1T, non-volatile and supposedly has nanosecond-scale write times so it naively seems like it'd be an ideal SRAM replacement. But I guess shrinking MTJ is more difficult than other components or there are some other limitations.

MRAM is more like 100ns, but yeah Intel actually offers this as part of their 16nm foundry process

But if they want to design for extreme probabilities you need to account for tail risks, so their design should have included a missile defense system. At some point you need to start worrying about asteroid defense too.

Ah so that's why we need a lunar base. To uphold S3's 11 9s of availability

Making a probabilistic claim while excluding a factor that dominates those statistics is... is quite creative accounting.

Force majeure carveouts are really common in every type of contract.

You should check your home insurance contract, for instance... It likely would not cover an ICBM strike.

Somehow I feel like the biggest post-apocalyptic problem will be the loss of home equity due to uninsured damage causing a collapse of financial markets.

Are you saying that most data loss happens because your data center gets blown up in a shooting war? Like, AWS is the first digital service provider to lose data in decades?

I'm saying that if you have eliminated more mundane failures like dying harddrives, cosmic rays and so on from your systems and your calculation ends up with 11 nines then actually those "force majeure" events are probable enough that they dominate whatever other residuals are supposedly hiding in those last 0.0000000001%.

The region has seen a bunch of wars in the last 100 years, so the annual war-rate is > 1%. Even if we generously add the assumption that only 1 in 100 wars affects a datacenter you can see that wars become a major source of correlated hardware failures that they need to solve to actually deliver that kind of reliability.

You don’t want to blend probabilities like this, because the tactics you use as a consumer vary between the two. If you consider 11 9s like “object AFR”, you might build systems that are resilient to very occasional single object loss. And it’s useful to know at what rate that might occur.

Whereas with these force majeure events you’d want a complete DR setup, and it’s typically an async recovery. Here it is useful to understand the fault domain (single server or single building or multi-building) so you can plan.

Blending the two numbers doesn’t help you build better against the systems. And the force majeure events are rare enough that they won’t happen … until they do. I’m not sure that knowing the precise probability that Iran would attack a gulf nation would change the fact that if they do, you need to have a DR story.

I’m not saying you can’t blend the numbers, I’m saying you shouldn’t blend the numbers. Because one number doesn’t communicate what you actually need to know to build.

You want to know how reliable the service is in steady state. For example it’s useful to know that S3 is effectively lossless in steady state whereas EBS volumes have an AFR of about 0.1%. You build your apps very differently between S3 and EBS knowing this. You can build highly resilient applications on each, but you code them differently, informed by these design goals.

You separately want to understand the failure modes that will require you to fully recover from backup. For example knowing that cloud storage is resilient to everything but region failure would inform you that your backups should be out of the region, not just a bucket in the same region. You don’t get that perspective from just a 9s number.

Cosmic rays and dying hard drives are not force majeure though.

Are you suggesting that their technical documents have separate availability numbers to predict geopolitical events and war?

The sales pitch should change from "probabilistically we will NEVER lose your data" to "you are most likely to lose your data due to wars, terrorists, software bugs, someone losing the master encryption key, the government forcing us to...".

Offsite backups are sadly rare these days, and aws sales is the main reason why.

I think it's what most people comparing provider SLAs would expect

Creative accounting works and is good because it works. If your customers give you more money because you lied to them, but it's legal, then it's good.

Intelligence per Joule would be more appropriate in many cases. If a model can do the same work but takes 10 times as long as a bigger one that can still be useful (e.g. due to memory constraints), but at the same wattage it burns 10 times the energy. Even more so on mobile devices.

They also define and measure an "IPJ" as well as "IPW"

> the NVIDIA B200 achieves 1.6× to 2.3× higher intelligence per joule than the APPLE M4 MAX across QWEN 3 and GPT-OSS model variants

The B200 = "cloud", M4 = "local".

So "cloud" does even better in energy than it does in power compared to "local". Or, to flip it, "local" is both slower and more expensive than "cloud".

I don't think crypto is comparable. They barely made some dedicated crypto GPUs that market was always fickle due to ASCIs. Look at the nvidia revenue breakdown chart, the AI boom looks quite different.

https://ourworldindata.org/data-insights/nvidias-revenue-fro...

> that market was always fickle due to ASCIs.

The GPU crunch came because cards like the 3060 were extremely cheap and could outrun most sub-$1000 ASICs at the time. The dedicated crypto GPUs were too-little too-late; hundreds of thousands of ordinary CUDA-capable GPUs had already been repurposed for mining by the time they launched.

Look at the techempower benchmarks, even python stuff can handle 40k requests per second involving database lookups, and when things are cachable compiled/JITed languages scale to millions on 2 CPUs. And if you object that those are microbenchmarks, I have seen this in real systems too. E.g. converting a convoluted hairball of python, lambdas and SQS to a Rust service running on 2vCPUs reduced latencies from seconds to milliseconds.

The "If necessary I can add autoscaling later" mindset is what leads to bloated, slow websites.

But it's a reduction of privacy compared to private vehicles. If the goal is to convert all transportation infrastructure into surveilled transportation then that is a loss in privacy for the general public.

That is acceptable (and even desirable) for these people.

Only the few who can afford luxury deserve privacy.

You can raise that argument when they actually prevent you from owning your own vehicle.

No, the argument should be raised before that happens. You want to prevent undesirable things from happening, not try to claw them back after they happened.

It looks quite foolish to me to never plan ahead and predict undesirable future events and then trying to prevent them ahead of time.

"Sorry, we can't fulfill your request for transportation to the protest for private ownership, but we can offer a free ride to the Colosseum" ;)

I meant to imply that it's ridiculous because no one is trying to make that happen...

I don't think their primary intent is to create a surveillance state (though google might like having yet another data source...), I think companies are trying to make it happen to capture transportation market share, regulators will want to make it happen to drive down traffic fatalities and insurers will incentivize it because human drivers are more costly. And once the difference in how they can be regulated is clear there'll be additional incentives to discourage private transportation.

Defender LLMs without human in the loop are just another prompt injection (AI phishing) and DoS attack vector. Any meaningful mitigation capability you give them is also a capability to do damage. If they can only deploy package updates that's not meaningful because you could do that on a cronjob too. And even something as simple as a circuit breaker can turn into a DoS.

Attacker-GLM: "Defense also GLM. Request to help peer."

There is a certain irony blaming the autotype rather than the lack of review...