I start to see a lot of these re-writes that depend on tests to state that its working. But the things that make software like Postgres and SQLite reliable are not mostly the test, but the real world production scars. That's where the reliability comes from, years and years of running in production.
> not mostly the test, but the real world production scars
Most extensive test suites are exactly production scars: every time you have a bug or a regression, you write a test that confirms correct behaviour.
SQLite is a good example to bring up because its extensive closed-source tests are what’s often cited as being what keeps people from forking it. (Turso did it, though, but it takes a company to deliver some guarantee of equivalent diligence.)
Sure, but behaviors that never have a bug or regression don't get a test. Software of this kind of complexity has all kinds of behavior that has never been broken, and doesn't have a specific test written for it.
Getting an extensive test suite passing is certainly orders of magnitude better than having no test suite at all, but it still doesn't tell you as much as you need to know. I would absolutely never trust an LLM Postgres rewrite (in any language) in production based on "only" Postgres's test suite passing.
I've also seen situations where a customer reports a bug, the fix breaks some regression, and the updated behavior to work around the fix breaking the regressions turns into an undocumented feature.
How do you break a regression? A regression is breakage. Are you one of the people who use "regression" to mean "regression test"? Did Codex learn this from you? I hate it.
The same basically holds for proofs in the absence of coherent global correctness criteria like, say, confluence and normalization for a lambda calculus, or soundness and completeness for a logic.
Fable's napkin estimate of the effort required to produce a passable reference semantics for Postgres, which would involve novel discoveries in denotational semantics of concurrent transactions and so on, might be in the ballpark of 30–60 years of PhD level work.
So realistically I think the only way to validate a Postgres implementation involves differential testing, fuzzing, acceptance test suites, etc. And still you'll have bugs that need to be hammered out the good old fashioned way.
Or even a human rewrite merely because some language is the current fad. A rewrite in a different language should be done for very good reasons, to solve problems that are bigger than the costs of all the bugs that will be introduced.
Perhaps before embarking on one of these rewrites the first step should be a heavy round of mutation testing and property based testing. Contribute any new testing code from this back to the original project. And *then* embark on the rewrite.
If that's your concern, then your argument becomes "software should never change". Why dare patch any bug ever? It might be load-bearing in some unknown, undocumented, unsupported workflow somewhere in the world. No test imaginable can catch that apart from the scream test.
There are reasonable arguments against language ports, but this is not one. You're making an argument against code changing at all ever.
The maintainers that wrote those tests will have experience you won't get out of a rewrite.
I think this is also where the real work is. A rewrite is one thing, that you can show off with a flashy blogpost. The maintenance, for years to come, won't be of that nature yet it still requires as much work.
This feels like the image of the plane that returns from battle with bullet holes, and the engineer being asked to path up where the holes to make it stronger. Only to be told to patch where there weren't holes as those planes didn't make it home.
While not an exact fit of an analogy, those tests patch what was a problem with Postgres in the wild. What it doesn't cover are the things that worked in Postgres without tests, but may fail in port and go undetected.
I don't necessarily disagree, but two other points to consider:
1. Every test that is written is another use case that wasn't tested before. 100% test coverage is often impractical, but the more tests you have the more of the code you can be confident about.
2. Every test you add is another regression that can't happen in the future; if you test the index rebuilding code and validate the output then you know that you aren't going to make a change that breaks the index rebuilding code. If you have a legitimate change you update the tests, but if you're not expecting the change then you know there's a bug somewhere.
Anything that doesn't have tests is unspecified behaviour. While it is true that a port may differ in behaviour where the behaviour is unspecified, "fail" is not the right framing as there is no definition of what it should do.
> Most extensive test suites are exactly production scars: every time you have a bug or a regression, you write a test that confirms correct behaviour.
If you can be 100% guaranteed that there indeed is a test for every occurred bug. Sometimes maintainers are not so strict about it.
And some programmers are so good that some issues are self-explanatory and they write good code to note a thing but don't write a test, because implementing the test is more expensive.
> And some programmers are so good that some issues are self-explanatory and they write good code to note a thing but don't write a test, because implementing the test is more expensive.
You don't write a test (just) to verify that your change fixed the issue, but to ensure it doesn't regress in the future after an unrelated refactor.
very naive. the runtime behavior of a rewrite should be significantly different in all kinds of unpredictable ways nobody see coming or might expect. It is a combination of language semantics, compiler behavior, operating system behavior, file system behavior, driver behavior, ..
yea, I found LLMs poor at reasoning through system-wide behaviors. Good luck to the vibers when they try to rootcause a rare data-corruption problem. They will hand it off to fable which will attempt trial-and-error bug fixes which will eventually muck up the code. What happens then ?
So you get other bugs when rewriting in another language without existing tests, got it. This is why I hate all the announcements of "it is rewritten in rust so it is obviously better than the original since it passes all the tests". Edit: and it's an LLM rewrite. Add that to the pile of over hyped messaging.
Unfortunately, too many people are getting captured by marketing and are divorcing themselves from reality. A rewrite can be an improvement, even if in the same or any other language.
But, there are also levels, in terms of quality and human code review, when dealing with rewrites. New bugs can be introduced or there can be style issues, that can take time to fully reveal themselves, and particularly if the person or people involved are not familiar with the other language.
So many comments here talking about the downsides. The only reason to do a rewrite is because there are massive upsides. Maybe the implicit point is that the upside (memory safety must be the biggest), isn't worth the downside (lots of bugs to be figured out before you trust it).
I find a lot of HN discussions quickly turn into thought experiments and philosophical debates that largely forget the original topic. For the most part, I find this idiosyncrasy charming and entertaining, but it does frequently result in forests being missed for the trees.
I agree. I also agree with the sibling reply that -
> every time you have a bug or a regression, you write a test that confirms correct behaviour.
What I fail to see in these rewrites however is - what about new bugs introduced by virtue of this rewrite? I mean it'll have to go through its own challenges in real-world scenarios, right?
> I start to see a lot of these re-writes that depend on tests to state that its working.
There's another way to validate the rewrite though. Just run both pgrust and postgres and compare the output. Know of an edge case? Run it too. Doesn't know? Use a fuzzer or some automated tool to find interesting inputs. Found an inconsistency? The input/output pair becomes a test case now
Not sure if there's tooling for that though. If there is, just give it to Claude so they will incorporate it in their development loop
Indeed. This approach is an improvement / augmentation over tests, not a panacea.
Neither postgres nor pgrust have their behavior specified using formal methods. (pgrust could write some contracts using something like kani or creusot, but having upstream postgres also write contracts is a tougher sell). If they had, one could write a giant proof that said the two software essentially do the same thing (at least in a subset of environments and some simplifying assumptions)
I can recommend proptest. What you're describing is a common pattern in property-based testing which basically boils down to "comparing against an oracle". In this case, postgres would be the oracle, pgrust is the system under test, and the idea is to generate strategies comprised of sequences of valid (and invalid) SQL statements and ensure the system under test behaves the same as the oracle in every case.
Not weighing in on this specific rewrite but tests are how you specify that your software works correctly. If a behavior isn’t covered by an automated test in some form you can’t assert that any given change doesn’t break it.
I think it is completely reasonable to use a preexisting unmodified test suite to state that something is working. The larger the project the more true this becomes. Real world production scars are documented and guarded against in the test suite otherwise those lessons get lost.
Also SQLite is legendary for its massive test suite and extensive fuzzing. They have 590x the amount of test code and scripts than normal code. Source: https://sqlite.org/testing.html
So, we should make it easier to feed that reliability back upstream.
Probably the most useful thing you can do with these LLM-transpilations for now: If the transpiled version passes all original tests, I can run my application test suite against it and use it to discover test coverage deficiencies in the original!
If it crashes or otherwise observably misbehaves, I know the real project was missing regression tests for something. We could make upstream so much more resilient against accidentally breaking stuff in future updates, if only it becomes safe (offline + no side effects) and easy (if it crashes/locks, it is not from some memory safety bug from 25k transactions earlier) to run these transpiled projects as one row in our everyday integration matrix.
As sibling mentioned - bugs and regressions are the thing that are (in a perfect world) usually covered.
The problem however is non-covered success cases. A visualisation of the problem: let's say universe of interaction for DB consists of 10.000 SQL queries. Over 10 years various regressions were found and 2.000 SQL queries are guarded by tests. In reference implementation remaining 8.000 never surfaced over this time and it's unclear if they will work.
And, thinking of how many various SQL queries PostgreSQL users around the world are using vs the test cases covered it's obvious that feature space isn't covered in 1% of the success ratio cases.
Now the new, test-based implementation, has to prove it can handle remaining 99%.
That's precisely what a regression test suite is for. There is a bug, you fix the bug, you add a regression test. So if the test suite is well maintained these real world production scars are reflected in the tests.
And also the amount of people running it in thousands of scenarios. Not sure if these areas can be even tested for, but I guess time will tell (can observe Bun if it breaks somewhere as that’s afaik the first big AI rewrite which got into prod for masses).
A lot of the signal (github, forums, mailing lists, discord, etc.) can be turned into signal. Right now it's easy enough to collect. In future it will be easy enough to cluster and generate preferences, experience, etc.
Every bug report, code change as a result, PR / commit message, PR comment that steers preferences, etc. is solid signal to generate future tests.
Software like a Database should have an extensive test bench with concurrency tests, all corner cases etc.
I'm not here running the new version on production to tell the maintainer/devs that my 'production unit tests failed'.
What is this even for logic?
I mean there is balance when i write tests for my production software, but my software is used by me. If i would have a library, i would test everything.
And there was some blog post about another database system were they even virtualized the File access to test cases like when the disk controller stops working.
The test suite is the result of these years of years of running in production. Every time you fix a bug, you add a non-regression test to ensure you don’t break it again.
In a project like PostgreSQL, those scars are reflected in unit tests demonstrating that they’re fixed. It’d be hard to pass its test suite and not be as robust as the original.
> It’d be hard to pass its test suite and not be as robust as the original.
This is not true, even in principle, even for Postgres itself. You'd be right to say that it'd be hard to pass the test suite and not be robust at all to some extent. But even in Postgres, I bet that you can quite easily introduce a change that will pass the whole test suite but reduce robustness compared to the latest release (for a somewhat silly example, add a call to `exit()` on a timer that's longer than the longest duration test in the suite - that will significantly reduce robustness while still passing the entire test suite).
Sure but these scars/tests are from the original implementation. Just because it doesn't have issues there doesn't mean it didn't bring its own set of issues
This is all well and good in theory, but the number of times I've seen tests that don't actually test what they say they're testing is hard to count. Yes even when you encourage the developers to ensure the test fails first and do TDD. Tests help you ship with confidence but there's usually at least a few that are just passing by pure luck.
So no, I wouldn't judge a rewrite as being equal just because it passes the tests. That said, I don't think that means you shouldn't do it. You just have to be pragmatic about it.
I dunno...I can envision something vibecoded prioritizing passing test suites producing something that does that, but isn't even functional in real-world production. Sort of like in the pre-AI world, where someone claims 'standards compliance' by way of passing compliance test suites, but can't actually interoperate well with other implementations of the standard. YMMV.
If you examine what you're saying here and slippery slope it a little, examining past effort for misses and correcting them at scale is not worth doing? There are many ways that the second part of your perspective are incorrect (burden / lack of point). I bet that a coding agent could in an automated fasion find at least reasonable additions that you'd say add value and reduce potential for error long term that you'd find valuable. They've probably already done so (I don't know postgres dev at all, so just supposition here. They will 100% do so in the future.
I think it's rather unfortunate that writing regression tests is seen as a heavy burden. Compared to determining the cause of a bug and the right strategy to fixing it in a maintainable way, writing such a test should be simple, no?
I'm not at ease regarding LLM generated code changes to a project with a (hopefully) long expected life time, but LLM generated regression tests should be less contentious. I wouldn't expect them to be maintained much; rather, if they don't perform as intended against a future build, just have them recreated.
Unit tests aren't useful for rewrites, only integration tests are. So there may be missing coverage. Also many things are simply difficult to test (eg performance under very specific conditions)
That's not relevant though. All concerns are secondary to security and Rust is the only language with security GUARANTEES. No other language is as secure. Therefore, even the worst Rust rewrite is automatically better than the best work in any other language, because it is the only one with guaranteed security.
If a Rust rewrite of any of your software becomes available and you aren't installing it immediately and without reservation, then you are simply not giving security the priority it both demands and deserves, and that makes you disastrously insecure. This is a serious issue that should be given all priority. There is no room for debate. Your only policies should be security before all else and compliance with those policies must be absolute and without deviation, or all is lost.
Also, there is more to security than memory errors. SQL injection, authentication, and access rules matter. It doesn’t matter if Rust database is secure to bad data if it lets anyone in to do anything. Or if it is crashing all the time or corrupting your data.
> If a Rust rewrite of any of your software becomes available and you aren't installing it immediately and without reservation
This is silly.
Rust is awesome, and it's hard to argue against in many domains. However, software is more than the language it is written in or the runtime serving it. Is the Rust rewrite fully compatible? Is it supported by a strong community? Is it likely to continue to be supported? Is its release cadence sensible? Is its licence compatible with your intended usage?
There are many questions needing to be answered before making rash decisions based purely on tech.
None of those concerns approach the level of priority that must be assigned to security. On defense, your security must be perfect forever or you are absolutely defeated. None of us are on the red team. It's not a rash decision, it's the only decision that logic allows. If you are not secure, you are NOTHING.
I strongly disagree. The easiest way to shut down your business is to insist on being 100% secure because the only way to have perfect security is to do nothing.
I think this is the way. Not control, but just make it simpler for parents to handle their childrens devices.
You dont have to make everyone share their age, you just make it so that parents can in a simpler way choose what the children should be able to access.
Make it easy to do right, dont add more control.
Its kind of the old anti-piracy copyprotections. The pirates always cracked it, and in the end, the ones who got to sit there trying to figure out what is the word in the manual is the user who actually paid for the game. So making it worse for the ones who paid, and better for the cracked version.
So, make it simple.
The tech community has had pretty much free reign over the last few decades, and has always chosen adult convenience over child safety (and mostly profit over both). The "middle ground" probably involves a bigger transfer than this.
There's probably a much better solution than "adults vs children" but very few with our expertise seem seriously interested in solving for safer children, which essentially leads to inexpert solutions gaining popular support.
I won't call myself an expert in this field, and haven't given it much thought, but a couple options just off the top of my head...
1. Limit child accounts to "classic" social network functionality. They get to see things from mutual friends. No algorithmic feeds, kids aren't in the user search, and no way for messages to be sent/received unless both sides have consented.
2. Disable chat for child accounts. How many chat apps do children really need? Each one is another potential vector for issues that parents would need to monitor.
I'm sure there is a monkey paw here, but either option seems better than no end-to-end encryption for anyone, at a time when government surveillance is a bigger issue than ever.
Frankly, I think option 1 would be better for all users, not just children. Go back to classic "social networks". This "social media" experiment has failed.
I feel like I can think of lots of situations where society puts in to protect children rather than leaving it to the parents (age ratings on films and games, YouTube Kids, regulations around advertising to children, the whole concept of school, reduced speed limits around playgrounds to give a few examples off the cuff).
My point more was that you said it wasn't a widely shared opinion, but to my mind, it is broadly the status quo. Whether that is good or bad is a separate point.
Its not reactionary to say you dont want the state to interfere too much in your child éducation.
Whether you are left or right its fine as long as the state aligns with you. But if you open an history book, you will sée it VERY OFTEN happened that states get crazy / ideological or just plain eugénist / liberticide.
My instinct is to say that I don't want the session as part of the commit. For me that is like a Slack thread discussing the new feature, and that is not something I would commit. I think that the split shouldn't be "is this done with a machine"=> commit, I think the split for AI should be the same as before. Is it code or changes of code, then it should be included. Is it discussing, going back and forth, that is not commited now.
On the other hand, if you do a plan that is then implemented, I actually do think it makes sense to save the plan, either as commit, or if you save that back to the issue.
But I actually think that one if the bigger arguments for single language models is the ability to have more languages. Im from Sweden, so I would like to have swedish on extremly high level, but I wouldnt like to have all other small languages on that level beacuse it would inflate the size. So, I actually think having multiple single language models, make it wider and deeper
Yes, I see the same thing. My working thesis is that if I can keep the codebase modular and clear seperations, so I keep the entire context, while claude code only need to focus on one module at a time, I can keep up the speed and quality. But if I try and give it tasks that cover the entire codebase it will have issues, no matter how you manage context and give directions. And again, this is not suprising, humans do the same, they need to break the task apart into smaller piecers. Have you found the same?
Yes. Spot on. The good thing is that it makes better code if modularity is strict as well.
I’m finding that I am breaking projects down into clear separations of concerns and designing inviolate API walls between modules, where before I might have reached into the code with less clearly defined internal vs external functions.
Exercising solid boundaries and being maniacal about the API surface is also really liberating personally, less cognitive load, less stress, easier tests, easier debugging.
Of course none of this is new, but now we can do it and get -more- done in a day than if we don’t. Building in technical debt no longer raises productivity, it lowers it.
If you are a competent engineer, ai can drastically improve both code quality and productivity, but you have to be capable of cognitively framing the project in advance (which can also be accelerated with ai). You need to work as an architect more than a coder.
