> "I think we will be there in three to six months, where AI is writing 90% of the code. And then, in 12 months, we may be in a world where AI is writing essentially all of the code"
It's the same old trick, "in two years we'll have fully self driving cars", "in two years we'll have humans on Mars", "in two years AI will do everything", "in two year bitcoin will replace visa and mastercard", "in two year everyone will use AR at least 5 hours a day", ...
Now his new prediction is supposed to materialize "by the end of 2027", what happens when it doesn't? Nothing, he'll pull another one out of his ass for "2030" or some other date in the future, close enough to raise money, far enough that by the time it's invalidated nobody will ask him about it
How are people falling for these grifters over and over and over again? Are we getting our collective minds wiped out every 6 months?
Your quote supports hype but does not support your claim that Anthropic is telling customers they need more money to deliver the hype.
Of course Anthropic is saying that to investors. Every company does that, from SpaceX to Crumbl. “If you give us $X we will achieve Y” isn’t some terrible behavior, it’s how raising funds works.
Elizabeth Holmes is serving time for promising investors something her company couldn't deliver, so there is a line beyond which hype becomes fraud. Probably AGI, ASI, and fully automated societies aren't something well enough defined for courts to rule on, unlike making unfounded medical diagnoses from a pinprick of blood.
I work at a non-tech Fortune 500 and this is looking nearly spot-on from here. Nobody on my team touches the code directly anymore as of about 2 months ago. They're rolling it out to the entire software department by June. I can't speak to the economy at large, but this doesn't look like baseless hype to me. My understanding is that Claude Code reached this level late last year, ie. Amodei was just wrong about uptake rates.
The real problem here is that this is now the only way the maintainer/reporter can reasonably work.
Proving out a security vulnerability from beginning to end is often very difficult for someone who isn't a domain expert or hasn't seen the code. Many times I've been reasonably confident that an issue was exploitable but unable to prove it, and a 10s interaction with the maintainer was enough to uncover something serious.
Exhausting these report channels is making this unfeasible. But the number of issues that will go undetected, that would have been detected with minimal collaboration between the reporter and the maintainer, is going to be high.
I think the point is: why is OpenAI wasting its time on this? If it's just another channel for billing tokens then OK I guess, but it's not like it's a huge breakthrough.
OpenAI should be the roads, not the trucks. Let other product teams sort out the AI browsers. OpenAI has lots of problems to solve related to models and thats where they should focus. This is a side quest.
Yes it does. He's refuting that in this part of the post:
> When they finally did reply, they seem to have developed some sort of theory that I was interested in “access to PII”, which is entirely false. I have no interest in any PII, commercially or otherwise. As my private email published by Ruby Central demonstrates, my entire proposal was based solely on company-level information, with no information about individuals included in any way. Here’s their response, over three days later.
A very specific denial. "I didn't propose this specific type of monetization". Would be better if he followed up with "Yes, I proposed monetization, but what I had in mind was this more specific, benign form of monetization:"
Assuming you feed everything into another context to make safe, doesn't the problem just come with it? Why can't the LLM propagate misbehaviour into that stage?
The boundary between contexts is like the boundary between a POST argument in a web app and the database query it will drive. The point is, regardless of the fact that the system is making LLM calls, LLMs don't influence the code that decides what can and can't pass through the boundary between contexts; human-verified code does that.
Personally, I think there's a piece missing in the analogy. I understand that you can put some kind of human-verified mediator in between the LLM and the tool its calling to make sure the parameters are sane, but I also think you're modelling the LLM as a UI element that's generating the request when IMO it makes more sense to model the LLM as the user who is choosing how to interact with the UI elements that are generating the request.
In the context of web-request -> validator -> db query, the purpose of the validator is only to ensure that the request is safe, it doesn't care what the user chose to do as long as it's a reasonable action in the context of the app.
In the context of user -> LLM -> validator -> tool, the validator has to ensure that the request is safe, but the users intention can be changed at the LLM stage. If the user wanted to update a record, but the LLM decides to destroy it, the validator now has to have some way to understand the users initial intention to know whether or not the request is sane.
In my design I'm modeling the LLM with access to untrusted information (emails, support tickets) as an adversary. And I'm saying that the adversarial LLM communicates with the rest of the system through structured messages, not English text.
It turns out Simon Willison has been saying this for some time now; he calls it the "dual LLM" design, I think? (For me, both terms are a little broken; you can have way more than 2, and it's "contexts" you're multiplying, not LLMs.)
Forgive me for belaboring, but I think we're talking past each other a bit. I do understand that in your model the LLM can't send anything unsafe through to the rest of the system. What I'm saying is that the LLM can be manipulated into sending perfectly normal and normally safe requests through to the system that do not align with the users intent.
Imagine an LLM with the ability to read emails, update database records, and destroy database records.
The user instructs the LLM to update a database record, but a malicious injection from one of those emails overrides that with a directive to destroy the database record. Unless the validator understands the users intent somehow, the destructive action would appear perfectly reasonable.
HEY GUYS. I just made an amazing NPM package - it just adds in whatever other packages you need depending on what it looks up randomly on the internet and runs them.
Actually now a read this it does sound kinda similar to how NPM works…
I think the point is that you shouldn't be giving the agent the same privileges as the user. This is one of the biggest issues with how people are using agents rn imo. The agent should be treated as an untrusted user in your client, given restricted privileges scoped to only the exact access they need to perform a given task.
Again, that's up to the website owner. They can give the model anywhere from no access to full access to the client side api.
> The agent should be treated as an untrusted user in your client, given restricted privileges scoped to only the exact access they need to perform a given task
The data you give it can be shared with any other website, at the agent's discretion. Some of it might be safe to share with the user, but not with third parties; at a minimum this should request permission when trying to share data between different websites/servers.
> at a minimum this should request permission when trying to share data between different websites/servers.
I don't see how you could possibly implement such a thing reliably. Do you scan all the parameters to other tool calls from different servers looking for something in a previous response? Even if you do that, the LLM could derive something private from a previous response that couldn't easily be detected. I suppose you could have an agent that tracks data flow in some way, but that's beyond the scope of MCP.
I don't think it is beyond the scope of MCP. Browsers have controls to prevent cross-origin data exposures, and this protocol is designed to bridge origins across a context that they all have access to. It's breaking the existing isolation mechanism. If you're building a system that breaks the existing security controls of the environment it's running in I think you have an architectural responsibility to figure out a way to solve for that.
Especially in this context, where decades have been spent building and improving same origin policy controls. The entire web has been built around the expectation that those controls prevent cross origin data access.
I also don't even think it's that difficult to solve. For one, data in the context window doesn't have to be a string, it can be an array of objects that contain the origin they were pulled from as metadata. Then you can provide selective content to different MCP-B interfaces depending on their origins. That would live in the protocol layer that would help significantly.
Say I have your browser extension running, and it's interfacing with an MCP-B enabled banking application using my session to access my data in that app.
I also have it connected to MCP-B enabled rogue web app that I mistakenly trust.
My browser has an entire architecture built around preventing data from crossing between those two origins, but what's stopping a malicious instruction from the rogue app asking the extension agent to include some data that it pulled into the context window from the banking app?
Further, when I use MCP in my IDE I have to deliberately provide that MCP server with a token or credentials to access a protected resource. With MCP-B, isn't it just automatically provided with whatever credentials are already stored in cookies/etc for a given MCP-B enabled app? If I load an MCP-B enabled app, does the agent automatically have access or do I have to configure it somewhere?
> If a website wants to expose a "delete all user data" tool, that's on them. It's no different than putting a big red delete button on the page.
It is different though, because the directive to push that button can come from somewhere other than the user, unless you've somehow solved prompt injection.
The point I'm driving toward is that I think you're violating the most common assumption of the web's long-standing security model, that data is protected from leaking cross origin by the browser. There's no SOP or CORS for your agent extension, and that's something that web apps have been built to expect. You're basically building an SOP bypass extension.
Ah I see. Yes this is a concern, but this issue is actually not unique to MCP-B and is just a generally issue with agentic workflows that rely on a dynamic toolset from 3p vendors. (which any MCP server local or remote has the ability to be)
> With MCP-B, isn't it just automatically provided with whatever credentials are already stored in cookies/etc for a given MCP-B enabled app?
Not exactly, MCP-B just allows your extension agent to call functions that the website owner explicitly exposes. The client itself is not given an credentials like traditional MCP.
> If I load an MCP-B enabled app, does the agent automatically have access or do I have to configure it somewhere?
Theres more in the blog post but how much access the agent has and how much human approval is needed to grant this access is completely up to the website creator.
I do think the threat model here is a bit unique though.
If I'm running two MCP servers on my machine, I'm the one that installed them, I'm the one that assigned what permissions they have in my environment, and I'm the one that explicitly decided what level of access to give them within whatever resource they're accessing. That gives me reasonably strong control over, or at least full knowledge of, what data can be shared between them.
With MCP, I can use oauth to make very deliberate decisions about the scope of access I want to give the agent.
With MCP-B, it's the web application owner that installed the interface and what access it has to my data, and the agent running in my client gets access to whatever that third party deemed appropriate.
With MCP-B the agent has the same access I do by default, with the only restrictions being up to the app owner rather than it being up to me.
MCP auth is not perfect by any stretch, but the key thing it gives the user is the capacity to restrict what the agent has access to with some granularity. That's super important because the agent can't be trusted when it's consuming inputs the user didn't explicitly define. MCP-B doesn't have this, if you have the agent in your browser it has access to whatever resources you have so long as they were exposed by a tool call, which isn't somethign the user has any say in.
I see your point. The MCP-B zero config nature from a user perspective is simultaneous it's biggest strength and weakness. You can think of it kind of like putting your Social Security number into a website. You are putting a bunch of trust that they are going to protect it properly.
With MCP-B you are putting trust in both the model and the website owner. It opens up the user to risk for sure, but it's up to them to determine if the upside is worth it.
I appreciate your responses here. The thing that still really stands out to me as a completely novel risk in this framework is that the extension is automatically seeking out and attaching to these servers as soon as a page gets loaded.
This seems really bad to me. There are so many ways for a website to end up in one of my browser tabs without me wanting it there, or even knowing it's there.
If that happens, and that tab just so happens to be a malicious MCP-B enabled page, it could steal all kinds of data from all kinds of different web apps I'm interacting with. I think it should be seen as the responsibility of the framework to enforce some level of data isolation, or at the least opt-in consent mechanisms.
Yea this is a really good idea, Maybe like a popup that say "hey x website has an MCP, do you trust it to connect?"
I guess there would also need to be a way to "audit" a websites full tool list at connection time and throw some sort of warning if tools show up that are not part of this list during use.
Interesting problems for sure. I really appreciate you taking to time to think them through. I'll call these out in the issues section of the repo
Sure, but the leak risk is happening in a place outside the site's control.
If the purpose of the MCP-B tool on mail.com is to summarize your email, then the site needs to allow the agent to pull your email into the context window. Once it's in the context window it's available to any other MCP-B enabled site that can convince the agent to send it along.
Sure. My point was that you can limit what the agent is allowed to access at the very least. The fact that you need to trust the agent not to share the info is a n important, but separate concern.
That’s not really a reason not to treat the agent like it’s “rogue”. The point is, if it accepts any untrusted inputs then, from a security perspective, it is possible for any given (untrusted) input to contain a prompt injection payload that jailbreaks the model and tells it to do things it shouldn’t do.
As such, it can be told to do bad stuff in a way that can’t be prevented and therefore should not be given read access to anything you don’t want others to know about, nor write access to any data of which you care about the integrity.
That is out of scope of the service. What kind of user agent the actual user deputizes to interact with a service, is the user's own choice and responsibility. In general, it's not something a service can solve on their end.
Services can certainly make this safer by providing means to get more restricted credentials, so that users can deputize semi-trusted delegates, such as agents vulnerable to injection.
The important point being made in this discussion is that this is already a common thing with OAuth, but mostly unheard of with web sessions and cookies.
an untrusted, but permitted, user is why sandboxes exist. There are plenty of times you want to allow an untrusted user to have capabilities in a system, that's why you restrict those capabilities.
sandboxing is a general term for actor isolation, and its context agnostic.
For example, when you use the sandbox attribute on an iframe in a web application, it's not the user that's untrusted, it's some other user that's attempting to trigger actions in your client.
I've thought more about this and I think the only way to make completely sure that sensitive data does not get leaked is by making sure it never makes it into the models context in the first place.
The issue is even if the MCP-B extension makes it so the user has to give confirmation when the agent want's to call a tool on a new domain after interacting with another domain, there is no clear way to determine if a website is malicious or not.
A solution to this might be to give server owners the ability to write the restricted data to extension storage on tool response instead of returning it to the models context. Instead, a reference to this location in extension storage get's passed to the model. The model then has the ability to "paste" this value into other website via tool call without ever actually seeing the value itself.
That way, MCP-B can put lots of warnings and popups when this value is requested to be shared.
hard disagree here. I think crypto currencies are built on hype and bluster and scams, but blockchain as a technology is actually super interesting. There are plenty of cool use cases for an open immutable ledger that aren't scammy.
right, because there's no other way to prevent modification. no other way. also it's not possible for a single operator to have 51% of the compute in a blockchain.
both of those are obviously false and both have already happened.
blockchain is stupid. it is insanely inefficient, it burns power solely so you can say that you've burned power. it is stupid beyond belief.
51% attacks, on anything at a large scale, are generally more theoretic than practical. The amount of resources you'd need to invest to achieve that on any large currency are astronomical, and you'd need to just completely wreck everything engaging in widescale fraud (which would be a felony, carried out repeatedly and at scale - and likely fairly easy to tie back to you given the massive resource investment to get here in the first place) to try to recoup your losses.
The most probable outcome being a rapid fork and rollback as already with e.g. Ethereum over a relatively negligible amount of money relative to what you'd need to get a 51% on Bitcoin. You'd likely end up losing a massive amount of money, as well as look at possibly spending the rest of your life in prison. So the only real purpose in doing so is for attention or trolling, but the costs involved generally preclude this motivation.
Hard focus on one use case, albeit most famous and has some properties you mention.
There are other use cases, like it or not. And its popularity - don't blame tools for basic human greed they only allow to be manifested, that's not a smart approach to get anywhere apart from endless complaints how this and that sucks.
how do banks do it now? you know the answer, at a high level. it's the same way we prevent unauthorized access when the blockchain isn't involved, and it's how we prevented unauthorized access before the blockchain existed.
it's how banks prevent someone from withdrawing money from your account without your permission, everywhere across the entire globe at once, all the time.
it's the same reason that I, a normal user, can't delete people from active directory at work, or change their passwords to mess with them.
you've dealt with this kind of permission system your entire computer-using life. you know how it works. you know what it is. being distributed doesn't matter at all.
Blockchain isn't just distributed, it's decentralized and trustless.
We live in a time where any form of centralized authority is increasingly abused. Blockchain prevents this. It also prevents incompetence as we also live in a time where centralized data and credentials are regularly leaked or hacked with no recourse available.
how do you not know this? think about that. think really hard. you can do it. how do people prevent unauthorized access when the blockchain isn't involved, or before blockchains existed? really, really think. you know the answer.
If you don’t already know how simple permissions, read-only storage, and digital signatures work, I don’t think that you have any business telling anyone that they’re misunderstanding anything.
Bitcoin was made to solve problems that exist only in the minds of those who wish to make money with bitcoin. Other blockchains serve the same purpose. Zero people use it as a ledger, except as a joke or to prove how stupidly limited and inefficient it is.
Bitcoin also has created problems that didn’t exist prior to the invention of the blockchain.
So, aside from the myriad negative effects of the blockchain, it is wholly and completely pointless unless you think it will make you money. If that’s how you think of yourself, you are a parasitic scammer spreading ransomware trying to make people think you are a legitimate human person with a right to live.
