I’m more worried that even if these tools do a bad job people will be too addicted to the convenience to give them up.
Example: recruiters locked into an AI arms race with applicants. The application summaries might be biased and contain hallucinations. The resumes are often copied wholesale from some chat bot or other. Nobody wins, the market continues to get worse, but nobody can stop either.
But why would the market get worse? Those recruiters aren’t just in an arms race with applicants, they’re also in an arms race with each other, and there’s an incentive to improve their tools too.
In fact, I use several. I have a series of journals for graphics programming. One for proof kernels. A series for projects... if I'm working on a program that takes me longer than a few hours to realize I keep notes, plans, etc.
This is the worst possible take. It dismisses an entire branch of science that has been studying neurology for decades. Biological brains exist, we study them, and no they are not like computers at all.
There have been charlatans repeating this idea of a “computational interpretation,” of biological processes since at least the 60s and it needs to be known that it was bunk then and continues to be bunk.
Update: There's no need for Chinese Room thought experiments. The outcome isn't what defines sentience, personhood, intelligence, etc. An algorithm is an algorithm. A computer is a computer. These things matter.
Neuroscience isn't a subset of computer science. It's a study of biological nervous systems, which can involve computational models, but it's not limited to that. You're mistaking a kind of map (computation) for the territory, probably based on a philosophical assumption about reality.
At any rate, biological organisms are not like LLMs. The nervous systems of human may perform some LLM-like actions, but they are different kinds of things.
But computational models are possibly the most universal thing there is, they are beneath even mathematics, and physical matter is no exception. There is simply no stronger computational model than a Turing machine, period. Just because you make it out of neurons or silicon is irrelevant from this aspect.
Turing machines aren't quantum mechanical, and computation is based on logic. This discussion is philosophical, so I guess it's philosophy all the way down.
Turing machines are deterministic. Quantum Mechanics is not, unless you go with a deterministic interpretation, like Many Worlds. But even then, you won't be able to compute all the branches of the universal wave equation. My guess is any deterministic interpretation of QM will have a computational bullet to bite.
As such, it doesn't look like reality can be fully simulated by a Turing machine.
Giving a Turing machine access to a quantum RNG oracle is a trivial extension that doesn't meaningfully change anything. If quantum woo is necessary to make consciousness work (there is no empirical evidence for this, BTW), such can be built into computers.
You would probably be surprised to learn that computational theory has little to no talk of "transistors, memory caches, and storage media".
You could run Crysis on an abacus and render it on board of colored pegs if you had the patience for it.
It cannot be stressed enough that discovering computation (solving equations and making algorithms) is a different field than executing computation (building faster components and discovering new architectures).
My point is that it takes more hand-waving and magic belief to anthropomorphize LLM systems than it does to treat them as what they are.
You gain nothing from understanding them as if they were no different than people and philosophizing about whether a Turing machine can simulate a human brain. Fine for a science fiction novel that is asking us what it means to be a person or question the morals about how we treat people we see as different from ourselves. Not useful for understanding how an LLM works or what it does.
In fact, I say it’s harmful. Given the emerging studies on the cognitive decline of relying on LLMs to replace skill use and on the emerging psychosis being observed in people who really do believe that chat bots are a superior form of intelligence.
As for brains, it might be that what we observe as “reasoning” and “intelligence” and “consciousness” is tied to the hardware, so to speak. Certainly what we’ve observed in the behaviour of bees and corvids have had a more dramatic effect on our understanding of these things than arguing about whether a Turing machine locked in a room could pass as human.
We certainly don’t simulate climate models in computers can call it, “Earth,” and try to convince anyone that we’re about to create parallel dimensions.
I don’t read Church’s paper on Lambda Calculus and get the belief that we could simulate all life from it. Nor Turing’s machine.
I guess I’m just not easily awed by LLMs and neural networks. We know that they can approximate any function given an unbounded network within some epsilon. But if you restate the theorem formally it loses much of its power to convince anyone that this means we could simulate any function. Some useful ones, sure, and we know that we can optimize computation to perform particular tasks but we also know what those limits are and for most functions, I imagine, we simply do not have enough atoms in the universe to approximate them.
LLMs and NNs and all of these things are neat tools. But there’s no explanatory power gained by fooling ourselves into treating them like they are people, could be people, or behave like people. It’s a system comprised of data and algorithms to perform a particular task. Understanding it this way makes it easier, in my experience, to understand the outputs they generate.
I don't see where I mentioned LLMs or what they have to do with a discussion about compute substrates.
My point is that it is incredibly unlikely the brain has any kind of monopoly on the algorithms it executes. Contrary to your point, a brain is in fact a computer.
> philosophizing about whether a Turing machine can simulate a human brain
Existence proof:
* DNA transcription (a Turing machine, as per (Turing 1936) )
* Leads to Alan Turing by means of morphogenisis (Turing 1952)
* Alan Turing has a brain that writes the two papers
* Thus proving he is at least a turing machine (by writing Turing 1936)
* And capable of simulating chemical processes (by writing Turing 1952)
>This is the worst possible take. It dismisses an entire branch of science that has been studying neurology for decades. Biological brains exist, we study them, and no they are not like computers at all.
They're not like computers in a superficial way that doesn't matter.
They're still computational apparatus, and have a not that dissimilar (if way more advanced) architecture.
Same as 0 and 1s aren't vibrating air molecules. They can still encode sound however just fine.
>Update: There's no need for Chinese Room thought experiments. The outcome isn't what defines sentience, personhood, intelligence, etc. An algorithm is an algorithm. A computer is a computer. These things matter.
Not begging the question matters even more.
This is just handwaving and begging the question. 'An algorithm is an algorithm' means nothing. Who said what the brain does can't be described by an algorithm?
> An algorithm is an algorithm. A computer is a computer. These things matter.
Sure. But we're allowed to notice abstractions that are similar between these things. Unless you believe that logic and "thinking" are somehow magic, and thus beyond the realm of computation, then there's no reason to think they're restricted to humanity.
It is human ego and hubris that keeps demanding we're special and could never be fully emulated in silicon. It's the exact same reasoning that put the earth at the center of the universe, and humans as the primary focus of God's will.
That said, nobody is confused that LLM's are the intellectual equal of humans today. They're more powerful in some ways, and tremendously weaker in other ways. But pointing those differences out, is not a logical argument in proving their ultimate abilities.
> Unless you believe that logic and "thinking" are somehow magic, and thus beyond the realm of computation
Worth noting that significant majority of the US population (though not necessarily developers) does in fact believe that, or at least belongs to a religious group for which that belief is commonly promulgated.
I think computation is an abstraction, not the reality. Same with math. Reality just is, humans come up with maps and models of it, then mistake the maps for the reality, which often causes distortions and attribution errors across domains. One of those distortions is thinking consciousness has to be computable, when computation is an abstraction, and consciousness is experiential.
But it's a philosophical argument. Nothing supernatural about it either.
You can play that game with any argument. "Consciousness" is just an abstraction, not the reality, which makes people who desperately want humans to be special, attribute it to something beyond reach of any other part of reality. It's an emotional need, placated by a philosophical outlook. Consciousness is just a model or map for a particular part of reality, and ironically focusing on it as somehow being the most important thing, makes you miss reality.
The reality is, we have devices in the real world that have demonstrable, factual capabilities. They're on the spectrum of what we'd call "intelligence". And therefore, it's natural that we compare them to other things that are also on that spectrum. That's every bit as much factual, as anything you've said.
It's just stupid to get so lost in philosophical terminology, that we have to dismiss them as mistaken maps or models. The only people doing that, are hyper focused on how important humans are, and what makes them identifiably different than other parts of reality. It's a mistake that the best philosophers of every age keep making.
The argument you're attempting to have, and I believe failing at, is one of resolution of simulation.
Consciousness is 100% computable. Be that digitally (electrical), chemically, or quantumly. You don't have any other choices outside of that.
Moreso consciousness/sentience is a continuum going from very basic animals to the complexity of humans inner mind. Consciousness didn't just spring up, it evolved over millions of years, and therefore is made up of parts that are divisible.
Reality is. Consciousness is.. questionable. I have one. You? I don't know, I'm experiencing reality and you seem to have one, but I can never know it.
Computations on the other hand describe reality. And unless human brains somehow escape the physical reality, this description about the latter should surely apply here as well. There are no stronger computational models than a Turing machine, ergo whatever the human brain does (regardless of implementation) should be describable by one.
Worth noting that this is the thesis of Seeing Red: A study in consciousness. I think you will find it a good read, even if I disagreed with some of the ideas.
The atoms of your body are not dynamic structures, they do not reengineer or reconfigure themselves in response to success/failure or rules discovery. So by your own logic, you can not be intelligent, because your body is running on a non-dynamic structure. Your argument lacks an appreciation for higher level abstractions, built on non-dynamic structures. That's exactly what is happening in your body, and also with the software that runs on silicon. Unless you believe the atoms in your body are "magic" and fundamentally different from the atoms in silicon; there's really no merit in your argument.
>>he atoms of your body are not dynamic structures, they do not reengineer or reconfigure themselves in response to success/failure or rules discovery.<<
you should check out chemistry, and nuclear physics, it will probably blow your mind.
it seems you have an inside scoop, lets go through what is required to create a silicon logic gate that changes function according to past events, and projected trends?
You're ignoring the point. The individual atoms of YOUR body do not learn. They do not respond to experience. You categorically stated that any system built on such components can not demonstrate intelligence. You need to think long and hard before posting this argument again.
Once you admit that higher level structures can be intelligent, even though they're built on non-dynamic, non-adaptive technology -- then there's as much reason to think that software running on silicon can do it too. Just like the higher level chemistry, nuclear physics, and any other "biological software" can do on top of the non-dynamic, non-learning, atoms of your body.
>>The individual atoms of YOUR body do not learn. They do not respond to experience<<
you are quite wrong on that. that is where you are failing to understand, you cant get past that idea.
there is also a large difference in scale. your silicon is going to need assembly/organization on the scale of individual molecules, and there will be self assembly required as that level of organization is constantly changing.
the barrier is mechanical scale construction, as the basic unit of function,that is why silicon and code cant adapt, cant exploit hysterisis, cant alter its own structure and function at an existentially fundamental level.
you are holding the wrong end of the stick. biology is not magic, it is a product of reality.
No, you're failing to acknowledge that your own assertion that intelligence can't be based on a non-dynamic, non-learning technology is just wrong. And not only wrong, proof to the contrary, is demonstrated by your very own existence. If you accept that you are at the very base of your tech stack, just atoms, then you simply must acknowledge that intelligence can be built on top of a non-learning, non-dynamic base technology.
All the rest is just hand waiving that it's "different". You're either atoms, or you're somehow atoms + extra magic. I'm assuming you're not going to claim that you're extra magic, in which case, your assertions are just demonstrably false, and predicated on unjustified claims about the nature of biology.
so you are a bot! i thought so, not bad, your getting better at acting human!
atoms are not the base of stack, you need to look at virtual annihilation, and decoherence. to get close to base. there is no magic, biology just goes to the base of the stack.
you cant access that base, with such coarse mechanisms as deposited silicon.
thats because it never changes, it fails at times and starts over.
biology is constantly changing, its tied to the base of existence itself. it fails, and varies until failure is an infeasible state.
Quantum "computers" are something close to where you need to be, and a self assembling, self replenishing, persistant ^patterning^ constraint is going to be of much greater utility than a silicon abacus.
sorry, but you are absolutely wrong on that one, you yourself are absolute proof.
not only that code is only as dynamic as the rules of the language will permit.
silicon and code cant break the rule, or change the rules, biological adaptive hysteretic, out of band informatic neural systems do, and repeat, silicon and code cant.
Programming languages are turing complete...the boundary is mathematics itself.
Unless you are going to take the position that neural systems transcend mathmatics (i.e. they are magic), there is no theoretical reason that a brain can't run on silicon. It's all just numbers, no magic spirit energy.
We've had evolutionary algorithms and programs that self-train themselves for decades now.
mathematics, has a problem with uncertainties, and that is why math, as structured cant do it. magic makes a cool strawman, but there is no magic, you need to refine your awareness of physical reality. solid state silicon wont get you where you want to go. you should look at colloidal systems [however that leads to biology] or if energetic constraints are not an issue, plasma state quantum "computation".
also any such thing that is generated, must be responsive to consequences of its own activities, capable of meta-training, rather than being locked into a training programming. a system of aligned, emergent outcomes.
Worth separating “the algorithm” from “the trained model.” Humans write the architecture + training loop (the recipe), but most of the actual capability ends up in the learned weights after training on a ton of data.
Inference is mostly matrix math + a few standard ops, and the behavior isn’t hand-coded rule-by-rule. The “algorithm” part is more like instincts in animals: it sets up the learning dynamics and some biases, but it doesn’t get you very far without what’s learned from experience/data.
Also, most “knowledge” comes from pretraining; RL-style fine-tuning mostly nudges behavior (helpfulness/safety/preferences) rather than creating the base capabilities.
> Biological brains exist, we study them, and no they are not like computers at all.
Technically correct? I think single bioneurons are potentially Turing complete all by themselves at the relevant emergence level. I've read papers where people describe how they are at least on the order of capability of solving MNIST.
So a biological brain is closer to a data-center. (Albeit perhaps with low complexity nodes)
But there's so much we don't know that I couldn't tell you in detail. It's weird how much people don't know.
Obviously any kind of model is going to be a gross simplification of the actual biological systems at play in various behaviors that brains exhibit.
I'm just pointing out that not all models are created equal and this one is over used to create a lot of bullshit.
Especially in the tech industry where we're presently seeing billionaires trying to peddle a new techno-feudalism wrapped up in the mystical hokum language of machines that can, "reason."
I don't think the use of the computational interpretation can't possibly lead to interesting results or insights but I do hope that the neuroscientists in the room don't get too exhausted by the constant stream of papers and conference talks pushing out empirical studies.
> There have been charlatans repeating this idea of a “computational interpretation,” of biological processes since at least the 60s and it needs to be known that it was bunk then and continues to be bunk.
I do have to react to this particular wording.
RNA polymerase literally slides along a tape (DNA strand), reads symbols, and produces output based on what it reads. You've got start codons, stop codons, state-dependent behavior, error correction.
That's pretty much the physical implementation of a Turing machine in wetware, right there.
And then you've got Ribosomes reading RNA as a tape. That's another time where Turing seems to have been very prescient.
And we haven't even gotten into what the proteins then get up to after that yet, let alone neurons.
So calling 'computational interpretation' bunk while there's literal Turing machines running in every cell might be overstating your case slightly.
To the best of our knowledge, we live in a physical reality with matter that abides by certain laws.
So personal beliefs aside, it's a safe starting assumption that human brains also operate with these primitives.
A Turing machine is a model of computation which was in part created so that "a human could trivially emulate one". (And I'm not talking about the Turing test here). We also know that there is no stronger model of computation than what a Turing model is capable of -> ergo anything a human brain could do, could in theory be doable via any other machine that is capable of emulating a Turing machine, be it silicon, an intricate game of life play, or PowerPoint.
It's better to say we live in a reality where physics provides our best understanding of how that fundamental reality behaves consistently. Saying it's "physical" or follows laws (causation) is making an ontological statement about how reality is, instead of how we currently understand it.
Which is important when people make claims that brains are just computers and LLMs are doing what humans do when we think and feel, because reality is computational or things to that effect.
There are particular scales of reality you don't need to know about because the statistical outcome is averaged along the principle of least action. A quantum particle could disappear, hell maybe even an entire atom. But any larger than that becomes horrifically improbable.
I don't know if you've read Permutation City by Greg Egan, but it's a really cool story.
Do I believe we can upload a human mind into a computing machine and simulate it by executing a step function and jump off into a parallel universe created by a mathematical simulation in another computer to escape this reality? No.
It's a neat thought experiment but that's all it is.
I don't doubt that one day we may figure out the physical process that encodes and recalls "memories" in our minds by following the science. But I don't think the computation model, alone, offers anything useful other than the observation that physical brains don't load and store data the way silicon can.
Could we simulate the process on silicon? Possibly, as long as the bounds of the neural net won't require us to burn this part of the known universe to compute it with some hypothetical machine.
That's a very superficial take. "Physical" and "reality" are two terms that must be put in the same sentence with _great_ care. The physical is a description of what appears on our screen of perception. Jumping all the way to "reality" is the same as inferring that your colleague is made of luminous RGB pixels because you just had a Zoom call with them.
And telling us how to do our jobs. As if they've ever compared the optimized output of clang and gcc on an example program to track down a performance regression at 2AM.
That, and also I specifically loathe the way AI has been created.
In my heart I’m a code “artist,” and like all artists that have been more directly attacked, I also feel personally attacked by all of my stolen “art” that is now monetized by big corporations that do not give a single f*ck about beauty in the code, or whatever.
This may be a strange position, idk. Anyways, that’s the reason why.
Enough curiosity to have read and understood the papers, formalize some of the more grand theorems (they’re not that impressive once restated this way), and listen to the experiences of developers who’ve adopted it.
That’s more than 3 minutes. I’m not being glib when I say I’ve given it serious thought.
The problem is that if you don’t understand the code you’re taking a risk shipping that code.
There’s a reason why most vibe coded apps I’ve seen leak keys and have basic security flaws all over the place.
If you don’t know what you’re doing and you’re generating code at scale that you can’t manage you’re going to have a bad time.
The models are trained on all the slop we had to ship under time pressure and swore we’d fix later, etc. They’re not going to autocomplete the good code. They’re going to autocomplete the most common denominator code.
I don’t agree that design is cheap. Maybe for line-of-business software that doesn’t matter much.
I often recommend developers learn and use at least one model checking language for this purpose. They’re really good at helping us reason about global properties of systems.
This isn’t unique to Haskell or functional programming at all. I’ve seen programmers from JavaScript to C++ suffer the same fate: tied into the boundaries of a single process they try to verify the correctness of their systems informally. By writing documents and holding long meetings or convincing themselves by prototyping. It’s only a matter of time before the find the error in production.
Hopefully it’s of no real consequence. The error gets fixed and it’s a week or two of extra work. But some times it triggers an outage that causes SLO’s to overshoot and SLAs to trigger which cost the company money. That outage could have potentially been avoided.
But it’s tough getting developers to see the benefit in model checking or any kind of formal methods. The hubris is often too great. It’s not just functional programmers! Although it surprised me when I was working with Haskell developers that so many were resistant to the idea.
(If one has experience and understands functional programming, learning TLA should be rather straight forward)
I also took a stab at the versioning challenge back in 2019 and wrote a library with a friend in a fevered evening for type safe migrations in Haskell [0].
In practice it’s a bit tricky to use because it doesn’t enforce encoding the schema identifier in the serialized payload. Oops.
But between that, automated typing from examples, and refinement it might be possible to develop a system that can manage to validate the serialized data against the parser and have the serialization data appear in the type. At that point you could query the whole system.
I’ve heard this metaphor before and I don’t think it works well.
For one, a power tool like a bandsaw is a centaur technology. I, the human, am the top half of the centaur. The tool drives around doing what I tell it to do and helping me to do the task faster (or at all in some cases).
A GenAI tool is a reverse-centaur technology. The algorithm does almost all of the work. I’m the bottom half of the centaur helping the machine drive around and deliver the code to production faster.
So while I may choose to use hand tools in carpentry, I don’t feel bad using power tools. I don’t feel like the boss is hot to replace me with power tools. Or to lay off half my team because we have power tools now.
There's DeWALT, Craftsman, Stanley, etc carpentry/mechanic power tool brands who make a wide variety of all manner of tools and tooling; the equivalents in computers (at least UNIXy) are coreutils (fileutils, shellutils, and textutils), netpbm, sed, awk, the contents of /usr/bin, and all their alternative, updated brands like fd, the silver searcher, and ripgrep are, or the progression of increased sharpening in revision control tools from rcs, sccs, svn, to mercurial and git; or telnet-ssh, rcp-rsync, netcat-socat. Even perl and python qualify as multi-tool versions of separate power tools. I'd even include language compilers and interpreters in general as extremely sharp and powerful power multi-tools, the machine shop that lets you create more power tools. When you use these, you're working with your hands.
GenAI is none of that, it's not a power tool, even though it can use power tools or generate output like the above power tools do. GenAI is hiring someone else to build a bird house or a spice rack, and then saying you had a hand in the results. It's asking the replicator for "tea, earl grey, hot". It's like how we elevate CEOs just because they're the face of the company, as if they actually did the work and were solely responsible for the output. There's skill in organization and direction, not all CEOs get undeserved recognition, but it's the rare CEO who's getting their hands dirty creating something or some process, power tools or not. GenAI lets you, everyone, be the CEO.
Why else do you think I go to work everyday? Because I have a “passion” for sitting at a computer for 40 hours a week to enrich private companies bottom line or a SaaS product or a LOB implementation? It’s not astroturfing - it’s realistic
Would you be happier if I said I love writing assembly language code by hand like I did in 1986?
My analogy is more akin to using Google Maps (or any other navigation tool).
Prior to GPS and a navigation device, you would either print out the route ahead of time, and even then, you would stop at places and ask people about directions.
Post Google Maps, you follow it, and then if you know there's a better route, you choose to take a different path and Google Maps will adjust the route accordingly.
Google Maps is still insanely bad for hiking and cycling, so I combine the old-fashioned map method with an outdoor GPS onto which I load a precomputed GPX track for the route that I want to take.
I think this argument would work if hand-written code would convey some kind of status, like an expensive pair of Japanese selvage jeans. For now though, it doesn't seem to me that people paying for software care if it was written by a human or an AI tool.
You only feel that way about power tools because the transition for carpentry happened long ago. Carpenters viewed power tools much as we do LLMs today. Furniture factories, equivalent of dark agentic code factories, caused much despair to them too.
Humans are involved with assembly only because the last bits are maniacally difficult to get right. Humans might be involved with software still for many years, but it probably will look like doing final assembly and QA of pre-assembled components.
To me, they all the same because they are all tools that stand between “my vision” and “it being built.”
e.g. when I built a truck camper, maybe 50% was woodworking but I had to do electrical, plumbing, metalworking, plastic printing, and even networking infra.
The satisfaction was not from using power tools (or hand tools too) — those were chores — it was that I designed the entire thing from scratch by myself, it worked, was reliable through the years, and it looked professional.
Maybe. I'm not sure its that different though? If one person can do the work of two because of power tools, then why keep both? Same with AI. How people feel about it doesn't seem relevant.
Maybe the right example is the role of tractors in agriculture. Prior to tractors you had lots of people do the work, or maybe animals. But tractors and engines eliminate a whole class of labor. You could still till a field by hand or with a horse if you want, but it's probably not commercially viable.
First, creating power tools didn’t cause mass layoffs of carpenters and construction workers. There continued to be a demand for skilled workers.
Second, power tools work with the user’s intent. The user does the planning, the measuring, the cutting and all the activities of building. They might choose to use a dovetail saw instead of fasteners to make a joint.
Third, programming languages are specifications given to a compiler to generate more code. A single programmer can scale to many more customers than a labourer using tools.
The classification of centaur vs reverse-centaur tools came to me by way of Corey Doctorow.
There might be ways to use the technology that doesn’t make us into reverse centaurs but we haven’t discovered that yet. What we have in its current form isn’t a tool.
Did power tools not cause layoffs? That seems like a dubious claim to me. Building a house today takes far fewer people than 100 years ago. Seems unlikely that all the extra labor found other things to do in construction.
The “work” is not creating for and while loops. The work for me is:
1. Looking at the contract and talking to sales about any nuances from the client
2. Talking to the client (use stakeholder if you are working for a product company) about their business requirements and their constraints
3. Designing the architecture.
4. Presenting the architecture and design and iterating
5. Doing the implementation and iterating. This was the job of myself and a team depending on the size of the project. I can do a lot more by myself now in 40 hours a week with an LLM.
6. Reviewing the implementation
7. User acceptance testing
8. Documentation and handover.
I’ve done some form of this from the day I started working 25 years ago. I was fortunate to never be a “junior developer”. I came into my first job with 10 years of hobbyist experience and implementing a multi user data entry system.
I always considered coding as a necessary evil to see my vision come to fruition.
It seems like you're doing a lot of work to miss the actual point. Focusing on the minutiae of the analogy is a distraction from the over arching and obvious point. It has nothing to do with how you feel, it has to do with how you will compete in a world with others who feel differently.
There were carpenters who refused to use power tools, some still do. They are probably happy -- and that's great, all the power to them. But they're statistically irrelevant, just as artisanal hand-crafted computer coding will be. There was a time when coders rejected high level languages, because the only way they felt good about their code is if they handcrafted the binary codes, and keyed them directly into the computer without an assembler. Times change.
In my opinion, it is far too early to claim that developers developing like it was maybe three years ago are statistically irrelevant. Microsoft has gone in on AI tooling in a big way and they just nominated a "software quality czar".
I used the future tense. Maybe it will be one hundred years from now, who knows; but the main point still stands. It would just be nice to move the conversation beyond "but I enjoy coding!".
I don’t think it’s correct to claim that AI generated code is just next level of abstraction.
All previously mentioned levels produce deterministic results. Same input, same output.
AI-generation is not deterministic. It’s not even predictable. And example of big software companies clearly show what mass adoption of AI tools will look like in terms of software quality. I dread if using AI will ever be an expectation, this will be level of enshittification never before imagined.
You're not wrong. But your same objection was made against compilers. That they are opaque, have differences from one to another, and can introduce bugs, they're not actually deterministic if you upgrade the compiler, etc. They separate the programmer from the code the computer eventually executes.
In any case, clinging to the fact that this technology is different in some ways, continues to ignore the many ways it's exactly the same. People continue to cling to what they know, and find ways to argue against what's new. But the writing is plainly on the wall, regardless of how much we struggle to emotionally separate ourselves from it.
They may not be wrong per se but that argument is essentially a strawman argument.
If these tools are non-deterministic then how did someone at Anthropic spend the equivalent of $20,000 of Anthropic compute and end up with a C compiler that can compile the Linux kernel (one of the largest bodies of C code out there).
To be frank, the C compilers source code were probably multiply times in its learning material, it just had to translate to Rust.
This aside, one success story doesn’t mean much, doesn’t even touch determinism question. Anthropic with every ad like this should have posted all the prompts they used.
People on here keep trotting out this "AI-generation is not deterministic." (more properly speaking, non-deterministic) argument on here …
And my retort to you (and them) is, "Oh yeah, and so?"
What about me asking Claude Code to generate a factorial function in C or Python or Rust or insert-your-language-of-choice-here is non-deterministic?
If you're referring to the fact that for a given input LLMs (or whatever) because of certain controls (temperature controls?) don't give the same outputs for the same inputs. Yeah, okay. If we're talking about conversational language that makes a meaningful difference to whether it sounds like an ELISA robots or more like a human. But ask an LLM to output some code then that code has to adhere to functional requirements independent of, muh, non-determinism. And what's to stop you (if you're so sceptical/scared) writing test-cases to make sure the code that is magically whisked out of nowhere performs as you so desire? Nothing. What's to stop you getting one agent to write the test-suite (and for you to review to the test-suite for correctness and for another agent to the write the code and self-correct based off of checking its code against the test-suite? Nothing
I would advise anyone encountering this but-they're-non-deterministic argument on HN to really think through what the proponents of this argument are implying. I mean, aren't humans non-deterministic. (I should have thought so.) So how is it, <extra sarcasm mode activated>pray tell</extra sarcasm mode activated> humans manage to write correct software in the first place?
I personally have jested many times I picked my career because the logical soundness of programming is comforting to me. A one is always a one; you don’t measure it and find it off by some error; you can’t measure it a second time and get a different value.
I’ve also said code is prose for me.
I am not some autistic programmer either, even if these statements out of context make me sound like one.
The non-determinism has nothing to do with temperature; it has everything to do with that fact that even at temp equal to zero, a single meaningless change can produce a different result. It has to do with there being no way to predict what will happen when you run the model on your prompt.
Coding with LLMs is not the same job. How could it be the same to write a mathematical proof compared to asking an LLM to generate that proof for you? These are different tasks that use different parts of the brain.
> A one is always a one; you don’t measure it and find it off by some error; you can’t measure it a second time and get a different value.
Linus Torvalds famously only uses ECC memory in his dev machines. Why? Because every now and again either a cosmic ray or some electronic glitch will flip a bit from a zero to a one or from a one to a zero in his RAM. So no, a one is not always a one. A zero is not always a zero. In fact, you can measure it and find it off by some error. You can measure it a second time and get a different value. And because of this ever-so-slight glitchiness we invented ECC memory. Error correction codes are a thing because of this fundamental glitchiness. https://en.wikipedia.org/wiki/ECC_memory
We understand when and how things can go wrong and we correct for that. Same goes for LLMs. In fact I would go so far as to say that someone doesn't even really think like how a software/hardware engineer ought to think if this is not nearly immediately obvious.
Besides the but-they're-not-deterministic crowd there's also the oh-you-find-coding-painful-do-you crowd. Both are engaging in this sort of real men write code with their bare hands nonsense -- if that were the case then why aren't we still flipping bits using toggle switches? We automate stuff, do we not? How is this not a step-change in automation? For the first time in my life my ideas aren't constrained by how much code I can manually crank out and it's liberating. It's not like when I ask my coding agent to provide me with a factorial function in Haskell it draws a tomato. It will, statistically speaking, give me a factorial function in Haskell. Even if I have never written a line of Haskell in my life. That's astounding. I can now write in Haskell if I want. Or Rust. Or you-name-it.
Aren't there projects you wanted to embark on but the sheer amount of time you'd need just to crank out the code prevented you from even taking the first step? Now you can! Do you ever go back to a project and spend hours re-familiarising yourself with your own code. Now it's a two minute "what was I doing here?" away from you.
> The non-determinism has nothing to do with temperature; it has everything to do with that fact that even at temp equal to zero, a single meaningless change can produce a different result. It has to do with there being no way to predict what will happen when you run the model on your prompt.
I never meant to imply that the only factor involved was temperature. For our purposes this is a pedantic correction.
> Coding with LLMs is not the same job. How could it be the same to write a mathematical proof compared to asking an LLM to generate that proof for you?
Correct, it's not the same. Nobody is arguing that it's the same. And it's wrong that it's different, it's just different that it's different.
> These are different tasks that use different parts of the brain.
> That's astounding. I can now write in Haskell if I want. Or Rust. Or you-name-it.
You're responsible for what you ship using it. If you don't know what you're reading, especially if it's a language like C or Rust, be careful shipping that code to production. Your work colleague might get annoyed with you if you ask them to review too many PRs with the subtle, hard-to-detect kind of errors that LLMs generate. They will probably get mad if you submit useless security reports like the ones that flood bug bounty boards. Be wary.
IMO the only way to avoid these problems is expertise and that comes from experience and learning. There's only one way to do that and there's no royal road or shortcut.
You’re making quite long and angry sounding comments.
If you’re making code in language you don’t know, then this code is as good as a magical black box. It will never be properly supported, it’s a dead code in the project that may do what it says it does or may not (a 100%).
I think you should refrain from replying to me until you're able to respond to the actual points of my counter-arguments to you -- and until you are able to do so I'm going to operate under the assumption that you have no valid or useful response.
Non-determinism means here that with same inputs, same prompts we are not guaranteed the same results.
This turns writing code this way into a tedious procedure that may not even work exactly the same way every time.
You should ask yourself, too: if you already have to spend so much time to prepare various tests (can’t trust LLM to make them, or have to describe it so many details), so much time describing what you need, then hand holding the model, all to get mediocre code that you may not be able to reproduce with the same model tomorrow - what’s the point?
I don't think he's missing the point at all. A band saw is an immutable object with a fixed, deterministic capability--in other words, a tool.
An LLM is a slot machine. You can pull keep pulling the lever, but you'll get different results every time. A slot machine is technically a machine that can produce money, but nobody would ever say it's a tool for producing money.
People keep trotting this argument out. But a band saw is not deterministic either, it can snap in the middle of a cut and destroy what you're working on. The point is, we only treat it like it's deterministic, because most of the time it's reliable enough that it just does what we want. AI technology will definitely get to the same level eventually. Clinging on to the fact that it isn't yet at that level today, is just cope, not a principled argument.
For every real valued function and every epsilon greater than zero, there’s a neural network (size unbounded) which approximates the function with precision epsilon.
It sounds impressive and, as I understand it, is the basis for the argument that algorithms based on NN’s such as LLM’s will be able to put perform humans at tasks such as programming.
But this theorem contains an ambiguous term that makes it less impressive when you remove it.
Which for me, makes such tools… interesting I guess for some applications but it’s not nearly as impressive as to remove the need for programmers or to replace their labour entirely with automation that we need to concern ourselves with writing markdown files and wasting tokens asking the algorithm to try again.
So this whole argument that, “you better learn to use them or be displaced in the labour market,” is a relying on a weak argument.
I think the distinction without a difference is a tool being deterministic or not. Fundamentally, its nature doesn't matter, if in actual practice it outperforms everything else.
Be that as it may, moving the goalpost aside. For me personally this fundamentally does matter. Programming is about giving instructions for a machine (or something mechanical) to follow. It matters a great deal to me that the machine reliably follows the instructions I give it. And compiler authors of the past have gone to great lengths to make their compilers produce robust (meaning deterministic) output, as have language authors tried to make their standards as rigorous (meaning minimize undefined behavior) as possible.
And for that matter, going back to the band saw analogy, a measure of a quality of a great band saw is, in fact, that the blade won’t snap in half in the middle of a cut. If a band saw manufacturer produces a band saw with a really low binomial p-value (meaning it is less deterministic/more stochastic) that is a pretty lousy band saw, and good carpenters will know to stay away from that brand of band saws.
To me this paints a picture of a distinction that does indeed have a difference. A pretty important difference for that matter.
I feel like we're both in similar minds of opposite sides, so perhaps you can answer me this: How is a deterministic AI any different from a search engine?
In other words, if you and me always get the same results back for the same prompt (definition of determinism,) isn't that just really, really power hungry Google?
I'm not sure pure determinism is actually a desirable goal. I mean, if you ask the best programmer in the world the same question every day, you're likely to eventually get a new answer at some point. But if you ask him, or I ask him, hopefully he gives the same good answer, to us both. In any case, he's not just a power hungry Google, because he can contextualize our question, and understand us when we ask in very obscured ways; maybe without us even understanding what we're actually looking for.
Have you never run a team of software engineers as a lead? Agentic coding comes naturally to a lot of people because that's PRECISELY what you do when you're leading a team, herding multiple brains to point them in the same direction so when you combine all their work it becomes something that is greater than the sum of it's parts.
Lots of the complains about agents sound identical to things I've heard and even said myself about junior engineers.
That said, there's always going to need to be people who can reach below the abstraction and agentic coding loops deprive you of the ability to get those reps in.
People say this about juniors but I've never seen a junior make some of the bone headed mistakes AI loves to make. Either I'm very lucky or other people have really stupid juniors on their teams lol.
Regardless, personally, there's no comparison between an LLM and a junior; always rather work with a junior.
I've wrote this a few times, but LLM interactions often remind me of my days at Nokia - a lot of the interactions are exactly like what I remember with some of their cheap subcons there.
I even have exactly the same discussion after it messed up, like "My code is working, ignore that failing test, that was always broking, and I definitey didn't break it just now".
> Have you never run a team of software engineers as a lead?
I expect juniors to improve fast to get really good. AI is incapable of applying the teaching that I expcect juniors to internalize to any future code that it writes.
Yes, I’ve read quite a lot about that bloody and terrible part of history.
The Ludddites were workers who lived in an era without any social or state protections for labourers. Capitalists were using child labour to operate the looms because it was cheaper than paying anyone a fair wage. If you didn’t like the conditions you could go work as an indentured servant for the state in the work houses.
Luddites used organized protests in the form of collective violence to force action when they had no other leverage. People were literally shot or jailed for this.
It was a horrible part of history written by the winners. That’s why everyone thinks Luddites were against technology and progress instead of social reforms and responsibility.
In that case I really don't understand how you conclude there's any difference between being on the bottom or the top of the tool. The bare reality is the same: Skilled labourers will be replaced by automation. Woodworking tools (and looms) replaced skilled labourers with less-skilled replacements (such as children), and AI will absolutely replace skilled labourers with less-skilled replacements as well. I ask sincerely, I truly don't understand how this isn't a distinction without a difference. Have you spent time inside a modern furniture factory? Have you seen how few people it takes to make tens of tons of product?
I haven’t worked in a furniture factory but I have assembled car seats in a factory for Toyota.
The difference matters because the people who worked together to smash the looms created the myth of Ned Ludd to protect their identities from persecution. They used organized violence because they had no leverage otherwise to demand fair wages, safety guarantees, and other labour protections. What they were fighting for wasn’t the abolishment of automation and looms. It was for social reforms that would have given them labour protections.
It matters today because AI isn’t a profit line on any balance sheet right now but it is being used to justify mass layoffs and to reduce the leverage of knowledge workers in the marketplace. These tools steal your work without compensation and replace your job with capital so that rent seekers can seek rent.
It’s not a repeat of what happened in the Luddite protests but history is rhyming.
We agree, which makes me question your original point with the power tool somehow being different even more. Every automation gives more leverage to capital over labour. That's the history of technology. Downstream it makes this great life with indoor plumbing etc possible but automation in any form will always erode skilled labourers as a class. It's all essentially the same in that regard.
The introduction of looms wasn’t what displaced workers.
It was capitalists seeking profits by reducing the power of labour to negotiate.
We didn’t mass layoff carpenters once we had power tools and automation.
We had more carpenters.
Just like we had more programmers once we invented compilers and higher level languages.
LLMs just aren’t like power tools. Most programming tools aren’t like power tools.
Programming languages might be close to being “power tools,” as they fit in the “centaur” category. I could write the assembly by hand or write the bash scripts that deploy my VMs in the cloud. But instead I can write a program, give it to a compiler, and it will generate the code for me.
LLM generated code fits in the reverse-centaur category. I’m giving it instructions and context but I’m not doing the work. It is. My labour is to feed the machine and deliver its output. If there was a way to remove me from that loop, you bet I’d be out of a job in a heartbeat.
> I don’t feel like the boss is hot to replace me with power tools. Or to lay off half my team because we have power tools now.
That has more to do with how much demand there is for what you're doing. With software eating the world and hardware constraints becoming even more visible due to the chips situation, we can expect that there will be plenty of work for SWE's who are able to drive their coding agents effectively. Being the "top" (reasoning) or the "bottom" half is a matter of choice - if you slack off and are not highly committed to delivering quality product, you end up doing the "bottom" part and leaving the robot in the driver's seat.
I’m more worried that even if these tools do a bad job people will be too addicted to the convenience to give them up.
Example: recruiters locked into an AI arms race with applicants. The application summaries might be biased and contain hallucinations. The resumes are often copied wholesale from some chat bot or other. Nobody wins, the market continues to get worse, but nobody can stop either.
reply