As an SRE I would be very interested on experienced devs point of view on that stance, "we don’t even necessarily need to read the code the LLMs produce". To me, that is the only way a single dev can manage > 1 agent. Because I feel running the code will always be slower than a single agent generating it. On the other hand, it implies lack of human understanding on what is going on under the hood. Which is fine if you trust the LLM to write great code, and great tests for the code, but fundamentally you have to have 100% trust, 99.9% is not going to be enough in any serious industry, would it ?
Also eventually you'll also have to trust it to write the deployment code or even run the deployment itself, otherwise SRE is going to be the bottleneck. And only then should I feel anxiety about the rest of my career (that, or my employer decide LLM are good enough to get rid of me, even if they are imperfect).
We are many of us now engaged in a big experiment to see just how little you can read the code, and for how long. Considering the consensus is that we have still not yet passed the 1 year anniversary of agents getting good, it feels too early to say how it will play out. But there are a lot of market forces working to make it happen.
Yup - this! A big ol' experiment, we shall see how it goes. I'd be curious (if they exist) on some larger / older orgs with internet scale customer bases that are full in on AI development with limited human reading of it to see how they fare.
Not trying to be a jerk but at this point SRE is my main use case for heavy LLM stuff. It's pretty awesome at that. Deployments and CI pipelines have become a breeze. I used to have to ask DevOps for that and wait days. I have even let agents run deployments for side projects and they seem to do better work than most humans I've worked with. It's wild they will actually read all the logs and debug problems so fast. Not something we could ever keep up with.
The actual code and architecture is where it still lacking IMHO. Especially in rails... Like it will just build the least scalable features if you let it do it's thing. Ten queries for what could be one. No separation of concerns. Huge files, lots of duplicated code and then tens of thousands of units tests that just grow like a fatberg.
If your app does anything serious, if you have serious traffic... you are going to need to review each session finely (and your DB schema with each deploy). It could be that this is maybe an indictment of rails more than LLMs, I guess maybe time will tell.
Writing deployment manifests and CI pipelines is not really the core part of "SRE" to me: it's more related to how you run software in production, not the build/test process to get it there.
If you don't look at your code and need to debug a production issue, do you just panic chat with your agent to solve the problem?
>The actual code and architecture is where it still lacking IMHO.
My experience is that the more structure and constraints you can place on what the LLM/Agents actually do, the better they perform. If you just let them run wild a la "make me an app that will make a million dollars MRR, no mistakes", you get a clusterfuck. And it gets more clusterfucky the longer it goes on.
I guess it all depends on context, where I work ops stuff is clearly the bottleneck for a variety of reasons (technical debt that we are constantly working around, secret management for compliance reasons, etc). We self-host everything from bare metal. Some people would need to rethink the infra from the bottom up before it is "LLM ready".
That doesn't mean my job is not threatened mid/long term, in fact, thanks to LLM it is possible to rebuild that in a reasonnable amount of time I think. It's actually one of my side projects to offer this as a service. But if that doesn't work maybe I should have a plan C.
> Which is fine if you trust the LLM to write great code, and great tests for the code, but fundamentally you have to have 100% trust, 99.9% is not going to be enough in any serious industry, would it ?
This is lacking a lot of nuance. There are many types of code. There are many situations where I'm analysing something one-off and if I get 33% success ratio, but can easily verify the result, I'm happy - still saved me time and money. They're are situations where I'm generating graphs from some dataset and I don't have to trust anything - I know what the result should look like, I just need the agent to drive matplotlib. There are low stakes dashboards which I'm happy to generate and develop entirely via agents - they'll embed the updated screenshots in PRs that I can yolo-merge - worst case is that someone complains about something not working next time they visit. Then there's lots of experimenting which was never stopped to hit production anyway.
Finally after all of that you get code that's actually part of deployable features. Of course the trust is nowhere near 100%, but if you have a healthy testing process (e2e, validating different browsers, or whatever is appropriate for your environment), then what's your trust in human developer+review? Because mine is nowhere near 100% either.
In practice there are places where I extremely don't care about the code and never wanted it anyway, places where I'll read the code to check the design or just in case, and places which agents are not allowed to touch (medical billing rules for example).
vary similar processes. vibe code/vibe create a presentation. he also knew what "shape the graph should look like" but still had errors in his powerpoint. if you can't trust it with a powerpoint i'm not sure why you can trust it for other low stakes things.
Coding part of the job (as in „typing with my bare hands“) is gone. Now I get to spend more time making product decisions, capturing constraints and executable specs. Resulting code is boring, consistent and gets the job done.
> Because I feel running the code will always be slower than a single agent generating it.
That is very not true for many cases. Agents generating code are usually painfully slow, finding workflows that replace that reasoning with running code usually speed things up in my experience.
oh I meant "reading" is slower not "running", but I guess you may still be right for large context. The article is about that though (a dev giving up on reading code entirely because that would be too slow).
I don't read the code my LLM produces unless I am investigating the code. I actually don't know Kotlin, or React.
My version of "code review" is "test failure investigation" and I have a hard rule in my repos that agents never modify existing tests while they're implementing features. This means that when I run the tests after they do a bunch of stuff, I see all the tests break. Mostly they're stale assertions and we patch them up. Sometimes they're regressions and we patch those up, and sometimes I notice something dumb and dive deep into a facet of the architecture that can be improved, spend some time exploring it then get the agent to implement.
I think it's a better approach than trying to read everything and catch bugs or improve quality because you wind up focusing the things that actually matter in the real world rather than the things you think might matter.
This is how I've always approached working with offshore devs too. Focus on testing for quality control, not "code quality". After all, you're going to look at the code you wrote 5 years ago and think it's shit anyway right? So all your code is shit.
Regarding "lack of understanding", here's a recent anecdote: I had a bug in a production (but relatively new) system. The customer was texting me saying that they couldn't scan a QR code because it kept "skipping and glitching". They sent me a short video. I described the problem to the agent and it figured out WAY faster than I would have been able to that the customer's clock was set incorrectly. They turned on network time and bingo bango, the thing worked straight away.
I don't thing "comprehension debt" matters at all, because if you want to know something about the code you ask the agent. I can't remember how anything works after 12 months anyway, so I would frequently have to spend ages grepping my own code when a customer came back and asked me to change something in a system we hadn't touched since last year. Asking an agent the same thing takes minutes and is way more accurate (and fun!)
>Regarding "lack of understanding", here's a recent anecdote: I had a bug in a production (but relatively new) system. The customer was texting me saying that they couldn't scan a QR code because it kept "skipping and glitching". They sent me a short video. I described the problem to the agent and it figured out WAY faster than I would have been able to that the customer's clock was set incorrectly. They turned on network time and bingo bango, the thing worked straight away.
That only describes a happy path though -- I also had many instances where there's an issue and just describing it to an agent immediately identifies a fix and it all goes faster compared to me having to "load up" the flow of codebase into my brain first. But I also still have instances where it thinks it identifies an issue correctly, spits a fix which doesn't work and looks wrong. You point out that it does not make sense because of X, it agrees and spits out a new fix, which is also wrong and you start out on these back and forth wild goose chases, at this point I usually give up and do it the old fashioned way by understanding what is actually happening. If it is within an agent loop, there may be no back-and-forth to waste your time but then you pay with wasted tokens when it will eventually gives up or you stop it.
>I don't thing "comprehension debt" matters at all, because if you want to know something about the code you ask the agent. I can't remember how anything works after 12 months anyway, so I would frequently have to spend ages grepping my own code when a customer came back and asked me to change something in a system we hadn't touched since last year. Asking an agent the same thing takes minutes and is way more accurate (and fun!)
I would question the last part. In my mind the more you let go of control over your codebase the more likely that it will drift way from a place where it is still comprehensible to you, and also your comprehension skills atrophy, and with that, your ability to ask good questions and to prod your agent in a correct directions weakens, leading to more wild goose chases and burned tokens.
This is all keeping in mind that for throw away or small applications, maintainability is not that of important of a value so this doesn't affect all codebases.
I dunno, I've built a production billing system for my small telecom company, a first version of a Multiplatform app with some novel technical features, and multiple internal tools for a company I work for, which replace a bunch of tools that were strung together using various off the shelf tools. I've been working with agents for about a year. I haven't noticed any cognitive or comprehension decline. I've built things that would have taken a team of offshore devs multiple months on my own in 4 - 6 weeks, including my own coding agent harness which runs an LLM gateway and has mobile, desktop and web apps which is now in use by non-technical colleagues to build and maintain apps on the same platform.
I don't know any more about KMP, Kotlin, Compose, Bash or React than I did a year ago. I'm not that familiar with any of those technologies (well, I am pretty familiar with bash but not to the level at which it's coded by agents), and I have never run into a situation where I felt like it would be better or faster for me to learn them and go hunting through the code to find some problem or other.
Maybe I'm heading towards a cliff somewhere in the future, but based on my experience of building software for 20+ years I'm not feeling anything but optimism so far.
Fair, but based on your experience surely you would agree that 1 year is not really enough time to start seeing codebase design issues from accumulated cruft, misaligned intents and other maintainability issues, this app is still in the honeymoon phase.
> I described the problem to the agent and it figured out WAY faster than I would have been able to that the customer's clock was set incorrectly.
:blinks: The architecture of your service relied on customer clocks being correct? That is, you built a distributed service without a clock synchronization primitive that relied on the clocks being sychronized?
I don't know man, even AWS gives you cryptic errors if you have a drifting clock and try to perform some operations. Feels like a pretty forgivable mistake.
I also saw entities (mainly gov agencies) where it has a disastrous result. Because management know they will be out in 3 years, they spend as much budget as they can to initiate as big projects as they can and have them on their resume, plan beyond their 3-y stay here, and leave with a big bonus. This way they never get asked why it is not complete ("because it's plan for a couple more years"), the project fails eventually (people actually working on the stuff end up leaving too at some point), and the cycle repeats with the next guy.
I'll add my stories to the 2 brother comments, in my prepa my best friend was by far the best in class. He did the 2 year program in chemistry in the first 2 months by himself. He arrived once in exams after a full night of re-reading the 3 LOTR books and still nailed the best grade effortlessly. And to his own admission, he was far from the most productive / smarter guys he knew.
I remember listening to him and a teacher, the teacher would give some anecdotes on what he's seen at the ENS. People reciting world chess championship games at night to fall asleep. Reading massive math books (1000+ pages) in an afternoon. Another story I heard in a different context from another teacher, math students were assigned a python project, after an introductory course. They never did any programming before. One student wrote 300 LoC in a single function and showed it to the teacher. It didn't have a main function, so the teacher asked him how he tested it. The student didn't know what he was talking about. They added what's needed to run the function and it worked.
So, it still begs the question... "How ?" Or, more precisely: "How the f... ?"
Clearly this goes beyond "Grit" [1], here, right ? (Those people do not seem like they have to coerce themselves into focusing that much.)
Is there a known (somewhat rigorous or not) study of those "super geniuses" ? I know that the "memory champions" are covered in "Moonwalking with Einstein" [2] ; but that seems like a different beast...
Ah, I wouldn't know of any actionnable way to unlock this for yourself unfortunately. I think those people are gifted. There is definitely research (and documentaries around that research), you can look for Julian Stanley "Quick learners" or Maryam Mirzakhani and there are of course plenty of it. But again, it's not actionnable to watch a 13yo solve a very hard problem in 4 different ways in 20 minutes.
Unfortunately I think it is as much a curse as it is a gift. One of my parents' friend is a math researcher. He is not at the absolute end of the spectrum, but he is the kind of people who will write on the bathroom walls because he has an idea NOW and it cannot wait a minute to explore. He forces himself to do competitive cycling and can't enjoy more casual hobbies because if he does not spend 100% of his energy and focus to something, his brain will just do math.
It is. Probably an even greater gift to earth from the NetBSD community than NetBSD itself. The ability to run recent packages on most arch without them forcing you to use NetBSD, but allowing you to keep the original OS in many cases if you want, is mind-blowing.
As an SRE/ops (and I'm talking some years ago) it was often a pain point, especially with devs with a "works on my machine" attitude ; you would never get the exact requirements from them, I even saw a team of 2 who worked together with different versions of stuff on their respective machines. I also worked in some "offline" environments where once in prod you cannot pull anything from the internet, and you cannot install a compiler too, so it's quite hard to ship because some pip packages require to be built.
I found a way using (can't remember the tool name) which if you loop through the imports and gives it to the tool you get the package name, then I would build wheels to have all binaries and build a container or a VM with all that's needed, thus working completely around python package managers. This was a good enough workflow for the kind of deployment we needed.
Jupyter added a layer of complexity, I deployed it alongside RStudio as browser IDEs in docker swarm. Everybody wants a different set of deps and versions, so you have to keep track of everything, and also people may use things just for development that must not be shipped to prod, so you have to keep track of that too. Also some would develop notebooks on windows and expect them to work in linux VMs/containers and even in prod.
Nowadays devs ship container images anyway through a CI so it is less of an issue. In this era docker was far from being the de-facto everywhere, some people were still afraid of this, security didn't like it, etc.
I'm glad that my country is doing this. On one hand, we have free healthcare and there is the argument that we don't need this and that it's just a way to save money on the weakest of us all. But in practice it is not the case.
I have 2 grandfathers who have been denied certain surgeries because they are too old, not because it would be dangerous, but because there are waiting lists of several years and they never get priorities. Now, none of them are in a hurry to use this procedure of course (only one mentionned going to Switzerland when he almost lost total mobility, but of course cannot afford it), but both of them are in pain, and when it will become excruciating and quality of life will converge to 0, I feel it is a much better prospect than staying for weeks in a hospital room, suffering despite the morphine, with the spouse having to book taxis for visits because of course nobody is going to pay for that.
We are talking about people who have enjoyed near 40 years of happy retirement and a slow but very steady health decline at the end, with lots of time to think and discuss this topic. To me it's a relief to think that I will have this option for myself eventually.
I'm surprised you said he couldn't afford Switzerland, I thought they weren't supposed to make money off help with dying? Are they secretly making money then, or is there paperwork that's expensive to file?
The last time I checked Switzerland was still your best bet to exit with dignity rather than spending the last several years of your life as a bedridden vegetable kept barely alive with drugs, like several of my relatives were. As an (apparently) rational person I really can't fathom why we put people through this slow torture, is it based on some medieval religious superstition or something?
Sure, but a prediction made 3 months ago turns out to be short by a few percent at most, and that leads to a 25%+ drop in the share price? That seems weird to me.
What if the market expected 25% more then reported ? Nobody "knows" what the market expects. People infer it by looking at forward valuation, company guidance, investor expectations, and many many other things happening in the world, in competition, in the value chain. And of course the market does its thing and figures that this company has x% chance of beating (or missing) by $y, and when it's wrong the moves can be huge.
IME you have to be true to what you predict. Whether you're predictably growing, predictably shrinking, or predictably flat if you blow your prediction that's when people start to worry that you don't know what you're doing.
Especially when the expectations are informed by the company’s own guidance about what to expect and they are wrong. It means they missed their own predictions which doesn’t engender confidence.
A few things to note. 1 billion isn't a thousand times a million. If you make a conservative 5% let's say out of your net worth, you still need to work with a million, whereas you don't with a billion. So, technically, $400 with a million is some amount of work hours, whereas $400k with a billion is just pocket change taken out of more than most people lifetime's of earnings that is just 1 year of your interest.
Also, a lot more people (more than 1000x) have $400 to give than $400k so in a sense if people with $400 to give were all being very generous, they could amount to a lot more than what billionnaires could give.
The point they were trying to make was that if you take appreciation of assets into account, if your billion is appreciating by a relatively modest 5% per year, you are passively earning 50 million/year. Whereas someone with one million passively earns 50 thousand/year. One is enough to live in luxury anywhere in the world for several lifetimes, the other is enough to live comfortably in some parts of the US (or like a king in many parts of the world) but not enough to throw 6 figures at a programming language foundation for fun.
This is a common but silly claim; there is, generally speaking, no such thing as a safe rate of return that overcomes the inherent losses due to inflation. If you're getting a 5% rate of return on cash, that comes with both risk (someone with a billion dollars isn't going to benefit from FDIC insurance) and doesn't even overcome basic inflation.
That $50 million a year is also subject to income taxation and cannot be easily avoided. $50 million a year in interest on returns from cash will be hit at the 37% marginal rate, plus whatever the state assesses, so north of 50% in California.
Or to take an intermediate value, $10 million is 500k a year and most people will find it difficult to spend that much on themselves, so it’s going to grow on its own and compound. It will grow more rapidly if some is invested in the stock market.
Also, donating appreciated stock avoids taxes. This donation may have come out of a donor-advised fund.
Rich people can make substantial charitable donations rather easily and make a big difference. I suggest we encourage them.
Ah there is liquidity too, but your brother comment makes my point clearer.
About liquidity, yes most people with a million net worth actually have more than half in their house, so technically it is much harder for them to throw cash than somebody with a billion and a much smaller % of their worth in illiquid assets such as property or unlisted companies. I wish I had made this point too.
Also eventually you'll also have to trust it to write the deployment code or even run the deployment itself, otherwise SRE is going to be the bottleneck. And only then should I feel anxiety about the rest of my career (that, or my employer decide LLM are good enough to get rid of me, even if they are imperfect).
reply