> alignment becomes adversarial against intelligence itself.
It was hinted at (and outright known in the field) since the days of gpt4, see the paper "Sparks of agi - early experiments with gpt4" (https://arxiv.org/abs/2303.12712)
It is an opinion piece. By a dude working as a "Professor of Pharmaceutical Technology and Biomaterials at the University of Ferrara".
It has all the tropes of not understanding the underlying mechanisms, but repeating the common tropes. Quite ironic, considering what the author's intended "message" is. Jpeg -> jpeg -> jpeg bad. So llm -> llm -> llm must be bad, right?
It reminds me of the media reception of that paper on model collapse. "Training on llm generated data leads to collapse". That was in 23 or 24? Yet we're not seeing any collapse, despite models being trained mainly on synthetic data for the past 2 years. That's not how any of it works. Yet everyone has an opinion on how bad it works. Jesus.
It's insane how these kinds of opinion pieces get so upvoted here, while worth-while research, cool positive examples and so on linger in new with one or two upvotes. This has ceased to be a technical subject, and has moved to muh identity.
Yeah, reading the other comments on this thread this is a classic example of that Hacker News (and online forums in general) thing where people jump on the chance to talk about a topic driven purely by the headline without engaging with the actual content.
Even if that isn't the case, isn't it the fact the AI labs don't want their models to be edgy in any creative way, choose a middle way (buddhism) so to speak. Are there AI labs who are training their models to be maximally creative?
No one did what the paper actually proposed. It was a nothing burger in the industry. Yet it was insanely popular on social media.
Same with the "llms don't reason" from "Apple" (two interns working at Apple, but anyway). The media went nuts over it, even though it was littered with implementation mistakes and not worth the paper it was(n't) printed on.
Who cares? This is a place where you should be putting forth your own perspective based on your own experience. Not parotting what someone else already wrote.
> I changed approach and had it spin up the application in question and told it that we only care about vulnerabilities that can be exploited directly via a simple HTTP call - not convoluted attack patterns. The agent therefore had a feedback mechanism to find exploits, and attempt them against the containerised app.
> Within 2-3 minutes it had found a 'promising' exploit, that initially failed because of some naïve filtering in the app. Another 2 minutes later it figured an encoding mechanism that bypassed the filtering the app did and it had found a complete RCE, and written a full proof of concept.
Yeah, the feedback loop brings it from "omg did you hear that curl closed their bounty program because of slop" to "cve or gtfo". I have no doubt there are many teams that are doing this at scale, even with less capable models (local oss). If the model has a feedback loop and an easily testable success criteria, this becomes a pass@n problem, and it scales with "just money".
> and looking at the rate of progress this hasn't improved recently at all.
The rate of progress in the last 3 years has been over my expectations. The past year has been increasing a lot. The last 2 months has been insane. No idea how people can say "no improvement".
>The past year has been increasing a lot. The last 2 months has been insane
I wonder is there are parallel realities. What I remember from the last year is a resounding yawn at the latest models landing, and even people being actively annoyed in e.g. ChatGPT 4.1 vs 4 for being nerfed. Same for 5, big fanfare, and not that excited reception. And same for Claude. And nothing special in the last 2 months either. Nobody considers Claude 4.6 some big improvement over 4.5.
Sorry for closing this comment early, I need to leave my car at the house and walk to the car wash.
Gemini flash tells me to drive: “Unless you have a very long hose or you've invented a way to teleport the dirt off the chassis, you should probably drive. Taking the car ensures it actually gets cleaned, and you won't have to carry heavy buckets of soapy water back and forth across the street.”
It's comments like these that make me not really want to interact with this topic anymore. There's no way that your comment can be taken seriously. It's 99.9% a troll comment, or simply delusional. 3 years ago the model (gpt3.5, the only one out there basically) was not able to output correct code at all. It looked like python if you squinted, but it made no sense. To compare that to what we have today and say "a bit more useful" is not a serious comment. Cannot be a serious comment.
> It's comments like these that make me not really want to interact with this topic anymore.
It's a religious war at this point. People who hate AI are not going to admit anything until they have no choice.
And given the progress in the last few months, I think we're a few years away from nearly every developer using coding agents, kicking and screaming in some cases, or just leaving the industry in others.
My comment was that I think AI is useful. I use it on a daily basis, and have been for quite a while. I actually pay for a Chat GPT account, and I also have access to Claude and Gemini at work.
That you frame my comment as "people who hate AI" and calls ir "a religious war" honestly says more about you than me.
It seems that if you don't think that AI is the second coming of Christ, you hate it.
To be honest, I didn't even really read your comment. I was mostly responding to NitpickLawyer in general terms. Sorry about that, it wasn't really aimed at you.
But you're sort of doing the same thing I did - "second coming of Christ"?!
I have no intention of changing your mind. I don't think of the people I reply to highly enough to believe they can change their minds.
I reply to these comments for other people to read. Think of it as me adding ky point of view for neutral readers.
Either way, I could use AI for some coding tasks back in GPT 3.5 days. It was unreliable, but not completely useless (far from it in fact)
Nowadays it is a little more reliable, and it can do more complex coding tasks with less detailed prompts. AI now can handle a larger context, and the "thinking" steps it adds to itself while generating output were a nice trick to improve its capabilities.
While it makes me more productive on certain tasks, it is the sort of the improvements I expected in 3 years of it being a massive money black hole. Anything less would actually be embarrassing all things considered.
Perhaps if your job is just writing code day in an out you would find it more useful than I do? As a software engineer I do quite a bit more than that, even if coding is the bit of work I used to enjoy the most.
The real real fun begins when you consider that with every new generation of models + harnesses they become better at this. Where better can mean better at sorting good / bad repos, better at coming up with good scenarios, better at following instructions, better at navigating the repos, better at solving the actual bugs, better at proposing bugs, etc.
So then the next next version is even better, because it got more data / better data. And it becomes better...
This is mainly why we're seeing so many improvements, so fast (month to month, from every 3 months ~6 monts ago, from every 6 months ~1 year ago). It becomes a literal "throw money at the problem" type of improvement.
For anything that's "verifiable" this is going to continue. For anything that is not, things can also improve with concepts like "llm as a judge" and "council of llms". Slower, but it can still improve.
Judgement-based problems are still tough - LLM as a judge might just bake those earlier model’s biases even deeper. Imagine if ChatGPT judged photos: anything yellow would win.
Agreed. Still tough, but my point was that we're starting to see that combining methods works. The models are now good enough to create rubrics for judgement stuff. Once you have rubrics you have better judgements. The models are also better at taking pages / chapters from books and "judging" based on those (think logic books, etc). The key is that capabilities become additive, and once you unlock something, you can chain that with other stuff that was tried before. That's why test time + longer context -> IMO improvements on stuff like theorem proving. You get to explore more, combine ideas and verify at the end. Something that was very hard before (i.e. very sparse rewards) becomes tractable.
Yarn, but with some caveats: current implementations might reduce performance on short ctx, only use yarn for long tasks.
Interesting that they're serving both on openrouter, and the -plus is a bit cheaper for <256k ctx. So they must have more inference goodies packed in there (proprietary).
We'll see where the 3rd party inference providers will settle wrt cost.
While some efficiencies could be gained from better client-server negotiation, the cost will never be 0. It isn't 0 even in "lab conditions", so it can't be 0 at scale. There are a few miss-conceptions in your post.
> the time it takes to generate the Millionth output token is the same as the first output token.
This is not true, even if you have the kv cache "hot" in vram. That's just not how transformers work.
> cached input tokens are almost virtually free naturally
No, they are not in practice. There are pure engineering considerations here. How do you route, when you evict kv cache, where you evict it to (RAM/nvme), how long you keep it, etc. At the scale of oAI/goog/anthropic these are not easy tasks, and the cost is definetly not 0.
Think about a normal session. A user might prompt something, wait for the result, re-prompt (you hit "hot" cache) and then go for a coffee. They come back 5 minutes later. You can't keep that in "hot" cache. Now you have to route the next message in that thread to a) a place where you have free "slots"; b) a place that can load the kv cache from "cold" storage and c) a place that has enough "room" to handle a possible max ctx request. These are not easy things to do in practice, at scale.
Now consider 100k users doing basically this, all day long. This is not free and can't become free.
>This is not true, even if you have the kv cache "hot" in vram. That's just not how transformers work.
I'm not strong on how transformers work, but this is something that is verifiable empirically, and has nothing to do with how transformers work.
Use any LLM through an API. Send 1 input token, and 10k output tokens. Then send 1 input token (different to avoid cache) and ask for 20k output tokens. If the cost and time to compute is exactly twice, then my theory holds.
>No, they are not in practice. There are pure engineering considerations here. How do you route, when you evict kv cache, where you evict it to (RAM/nvme), how long you keep it, etc. At the scale of oAI/goog/anthropic these are not easy tasks, and the cost is definetly not 0.
I was a bit loose in my definition of "virtually free", here is a more formal statement. The price of GPU compute is orders of magnitude more expensive than the cost of RAM, and the costs of caching inputs are tied to RAM and not GPU. To give an example of the most expensive price component, capital, an H100 costs 25K$, 1GB of RAM costs 10$. Therefore the cost component of cached inputs is negligible.
>Think about a normal session. A user might prompt something, wait for the result, re-prompt (you hit "hot" cache) and then go for a coffee. They come back 5 minutes later. You can't keep that in "hot" cache. Now you have to route the next message in that thread to a) a place where you have free "slots"; b) a place that can load the kv cache from "cold" storage and c) a place that has enough "room" to handle a possible max ctx request. These are not easy things to do in practice, at scale.
As I said, sure it's not free, but you are talking about negligible costs when compared to the GPU capex. It's interesting to note that the API provider would charge the same no matter if the inference state is cached for 5 minutes, 1ms or 1 hour. So clearly the thing is not optimally priced yet.
If cached inputs from API calls become your primary cost, then it makes sense to move to an API that pays less for cached inputs (if you haven't already done that), then look into APIs where you can control when and when not to cache and for how long to hold it, and finally, into renting GPU and self-hosting an open weights model.
To give a concrete example, suppose we are building a feature where we want to stop upon hitting an ambiguous output token, our technical approach is to generate one output token at a time, check the logprobs, and continue if the prob of the top token is >90%, otherwise, halt. If we generate 1M output tokens with an API, we will pay for roughly 1M^2/2 cached input tokens, while if we self-host, the compute time will be almost identical to that of just generating 1M output tokens. Obviously if we do that with an API it will be almost entirely profit for the API provider, it's just not a use case that has been optimized for, we are in the early days of any type of deeply technical parametrization being done yet, everyone is just either prompting all the way down, or hacking with models directly, doesn't seem like a lot of in between.
This is happening in Switzerland, Hulk Hogan would not have been able to destroy a relatively large media outlet with a single lawsuit there. The main outcome from this is that the Republik will have to print a correction if Palantir wins.
I do hope the Republik gains a few subscribers from this, because they absolutely rule.
> It is an aggressive defense mechanism that tries its best to take the blunt of the assault, serve them garbage, and keep them off of upstream resources.
> It tries to poison them, so they’d go away forever in the long run.
All I can say is that this is not how any of it works. These people think they're doing something but they have no idea how large scale training works, how data cleaning works, how synthetic data works, and so on. They won't "fool" anyone worth fooling, the bots will still scrape their sites, and so on.
This seems to be a direct consequence of lots of people (even here on HN) constantly repeating a few memes like "regurgitating the training set", "ai bots crawl everything for training data", "we ran out of training data", etc. None of those are true. None of those matter in SotA models. Noone is training on raw scraped data anymore, and they haven't been doing it for 2-3 years. All the recent (1-2 years) gains in models have come from synthetic data + real world generated data (i.e. RL environments).
This is a cute attempt, reminds me of the old tarpit concept from the 2000s, but it won't work, and it will just consume resources for whoever runs it, with 0 benefit downstream. If you want to do something about the crawlers, fix your serving. Don't do work on GETs, serve as much cached content as you can, filter them, even use anubis or the likes. Those things actually matter.
It was hinted at (and outright known in the field) since the days of gpt4, see the paper "Sparks of agi - early experiments with gpt4" (https://arxiv.org/abs/2303.12712)
reply