"Scrubs differ significantly from traditional filesystem checks. Tools such as fsck or chkdsk examine logical structures and attempt to repair inconsistencies related to directory trees, allocation maps, reference counts, and other metadata relationships. ZFS does not need to perform these operations during normal scrubs because its transactional design ensures metadata consistency. Every transaction group moves the filesystem from one valid state to another. The scrub verifies the correctness of the data and metadata at the block level, not logical relationships."
> ZFS scrubs do not check filesystem objects for correctness and consistency; it only checks that they have the expected checksum and so have not become corrupted due to disk errors or other problems
A scrub literally reads the object from disk. And, for each block, the checksums are read up the tree. The object is therefore guaranteed to be correct and consistent at least re: the tree of blocks written.
> Unfortunately it's possible for ZFS bugs and issues to give you filesystem objects that have problems
Can you give a more concrete example of what you mean? It sounds like you have some experience with ZFS, but "ZFS doesn't have an fsck" is also some truly ancient FUD, so you will forgive my skepticism.
I'm willing to believe that you request an object and ZFS cannot return that object because of ... a checksum error or a read error in a single disk configuration, but what I have never seen is a scrub that indicates everything is fine, and then reads which don't return an object (because scrubs are just reads themselves?).
Now, are things like pool metadata corruption possible in ZFS? Yes, certainly. I'm just not sure fsck would or could help you out of the same jam if you were using XFS or ext4. AFAIK fsck may repair inconsistencies but I'm not sure it can repair metadata any better than ZFS can?
> Can you give a more concrete example of what you mean?
There's been several instances. For example, the send/receive code has had bugs leading to cases[1] where the checksum and hence scrub look fine but the data is not.
edit: the recent block cloning has also had some issues, eg[2][3].
I'm pretty sure it's also possible for hardware errors like bad memory to cause the data to get corrupted but the checksum gets computed on the corrupted data, thus it looks ok when scrubbed.
I think you're missing the 2nd feature to the parent's point that I take issue with, which is this is not just a bug that a scrub wouldn't find, but it must also be a bug which an fsck would find.
The parent's point is -- ZFS should have an fsck tool because an fsck does something ZFS cannot do by other means. I disagree. Yes, ZFS has bugs like any filesystem. However, I'm not sure an fsck tool would make that situation better?
> I think you're missing the 2nd feature to the parent's point that I take issue with
You're right, I did gloss over that point.
I guess it should be noted that a lot of what fsck does[1] on say ext4 is something ZFS does on pool import, like replying the journal (ZIL) or trying older superblocks (uberblocks[2]). In that regard it's acting more like XFS[3] from what I can see, which just exits with 0.
Generally, it's possible to have data which is not corrupted but which is logically inconsistent (incorrect).
Imagine that a directory ZAP has an entry that points to a bogus object ID.
That would be an example.
The ZAP block is intact but its content is inconsistent.
Such things can only happen through a logical bug in ZFS itself, not through some external force.
But bugs do happen.
If your search through OpenZFS bugs you will find multiple instances.
Things like leaking objects or space, etc.
That's why zdb now has support for some consistency checking (bit not for repairs).
> Imagine that a directory ZAP has an entry that points to a bogus object ID. That would be an example. The ZAP block is intact but its content is inconsistent.
The above is interesting and fair enough, but a few points:
First, I'm not sure that makes what seems to be the parent's point -- that scrub is an inadequate replacement for an fsck.
Second, I'm really unsure if your case is the situation the parent is referring to. Parent seems to be indicating actual data loss is occurring. Not leaking objects or space or bogus object IDs. Parent seems to be saying she/he scrubs with no errors and then when she/he tries to read back a file, oops, ZFS can't.
The two obvious examples that come to mind are native encryption bugs and spacemap issues.
Nothing about walking the entire tree of blocks and checking hashes validates the spacemaps - they only come up when you're dealing with allocating new blocks, and there have been a number of bugs where ZFS panics because the spacemaps say something insane, so you wind up needing to readonly import or discard the ZIL because it panics about trying to allocate an already-allocated segment if you import RW - and if your ondisk spacemaps are inconsistent in a way that discarding the ZIL doesn't work around, you would need some additional tool to try and repair this, because ZFS has no knobs for it.
Native encryption issues wouldn't be noticed because scrubbing doesn't attempt to untransform data blocks - you indirectly do that when you're walking the structures involved, but the L0 data blocks don't get decompressed or decrypted, since all your hashes are of the transformed blocks. And if you have a block where the hash in the metadata is correct but it doesn't decrypt, for any reason, scrub won't notice, but you sure will if you ever try to decrypt it.
Appreciate this rincebrain. Know that you know better than most and this certainly covers my 2nd point. I don't imagine these cases cover my first point though? These are not bugs of the type a fsck would catch?
A fsck could pretty readily notice and repair the spacemap inconsistencies - zdb already generates its own spacemaps and compares to reality on import.
If you have the keys, technically nothing stops a fsck from noticing the encryption problems, but yes, usually it wouldn't unless you had some known issue you added special detection for, like when XFS years ago had problems with if you mounted it with inode64 once and then not the next time so the inode numbers would wraparound.
I think you may be misreading my point above. I am not arguing ZFS doesn't have bugs. That's nuts. I am arguing that the bug the parent says he has would be an extraordinary bug.
This is not just a bug that a scrub wouldn't find, but also it is a bug which an fsck would find. And it is not just a bug in the spacemaps or other metadata, but the parent's claim is this is a bug which a scrub, which is just a read, wouldn't see, but a subsequent read would reveal.
I am the parent, and any competent fsck should find these issues. Fsck traditionally explicitly verifies claimed free space against actual free space, and in a filesystem with ACLs it should also verify that filesystem level metadata like ACLs is sane, just as fsck verifies (for example) sane inode flags and inode field values. ZFS scrubs explicitly do not verify spacemap consistency or a lot of other sorts of consistency.
ZFS scrubs don't even verify that a filesystem's directory tree is acyclic and can reach every claimed in-use filesystem object, but I'm not aware of ZFS bugs in that area. This is because ZFS's 'metadata' for scrubs is much different than how it works in a traditional filesystem. To phrase it in conventional filesystem terms, ZFS has a big list of all in-use inodes, and it verifies the filesystem checksums by going through this list. The 'filesystem metadata' that a scrub verifies is the structure of this list of in-use inodes, plus some other things around the corners.
Ubuntu shipped with a bug that they introduced by way of a very badly done patch. While I get your point, I don't think it's fair to use Ubuntu as a source - they're grossly incompetent when it comes to handling ZFS.
Imagine a race condition that writes a file node where a directory node should be. You have a valid object with a valid checksum, but it's hooked into the wrong place in your data structure.
> Imagine a race condition that writes a file node where a directory node should be. You have a valid object with a valid checksum, but it's hooked into the wrong place in your data structure.
A few things: 1) Is this an actual ZFS issue you encountered or is this a hypothetical? 2) And -- you don't imagine this would be discovered during a scrub? Why not? 3) But -- you do imagine it would be discovered and repaired by an fsck instead? Why so? 4) If so, wouldn't this just be a bug, like a fsck, not some fundamental limitation of the system?
FWIW I've never seen anything like this. I have seen Linux plus a flaky ALPM implementation drop reads and writes. I have seen ZFS notice at the very same moment when the power dropped via errors in `zpool status`. I do wonder if ext4's fsck or XFS's fsck does the same when someone who didn't know any better (like me!) sets the power management policy to "min_power" or "med_power_with_dipm".
Here's an example: https://www.illumos.org/issues/17734. But it would not be discovered by a scrub because the hashes are valid. Scrubs check hashes, not structure. It would be discovered by a fsck because the structure is invalid. Fscks check structure, not hashes.
They are two different tools, with two different uses.
How is the structure not valid here? Can you explain to us how an fsck would discover this bug (show an example where an fsck fixed a similar bug) but ZFS could never? The point I take contention with is that missing an fsck is a problem for ZFS, so more specifically can you answer my 4th Q:
>> 4) If so, wouldn't this just be a bug, like (a bug in) fsck, not some fundamental limitation of the system?
So -- is it possible an fsck might discover an inconsistency ZFS couldn't? Sure. Would this be a fundamental flaw of ZFS, which requires an fsck, instead of merely a bug? I'm less sure.
You do seem to at least understand my general contention with the parent's point. However, the parent is also making a specific claim about a bug which would be extraordinary. Parent's claim is this is a bug which a scrub, which is just a read, wouldn't see, but a subsequent read would reveal.
So -- is it possible an fsck might discover this specific kind of extraordinary bug in ZFS, after a scrub had already read back the data? Of that I'm highly dubious.
> Can you show us how an fsck would discover this bug but ZFS could never?
I'd have to read closer to be certain, but if my understanding of it is correct, you'd have orphaned objects in the file system. The orphaned objects would be detectable, but would have correct hashes.
> if so, wouldn't this just be a bug, like (a bug in) fsck, not some fundamental limitation of the system?
It's not a bug or a fundamental limitation of the system, it's just that fsck != scrub, and nobody has written the code for fsck. If someone wanted to write the code, they could. I suspect it wouldn't even be particularly hard.
But fsck != scrub, and they catch different things.
> That will also put it on the unfortunate position of being the place that breaks every time somebody adds a bug to the C code.
Can someone explain charitably what the poster is getting at? To me, the above makes zero sense. If the Rust code is what is implemented correctly, and has the well-defined semantics, then, when the C code breaks, it's obviously the C code's problem?
I think a charitable interpretation is that given that the Rust code will be less forgiving, it will "break" C code and patterns that "used to work", albeit with latent UB or other nonobvious correctness issues. Now, obviously this is ultimately a good thing, and no developer worth their salt would seriously argue that latent bugs should stay latent, but as we've already seen, people have egos and aren't always exceedingly rational.
It's not exactly that the Rust implementation will be less forgiving. But developers will have a clear picture of the operations semantics, while C developers will have a harder time understanding the operations, and consequently avoid them.
You have it the wrong way around. Here, ZFS uses many small S3 objects as the storage substrate, rather than physical disks. The value proposition is that this should be definitely cheaper and perhaps more durable than EBS.
Yes that is the value prop. Cheap S3 instead of expensive EBS.
EBS limitations:
- Per-instance throughput caps
- Pay for full provisioned capacity whether filled or not
S3:
- Pay only for what you store
- No per-instance bandwidth limits as long as you have network optimized instance
>> Certainly, these companies not endure as innovators: when coercion is your business model, innovation is not merely unnecessary but actively antithetical.
Oracle and VMware do seem like just rent seekers. I'm sure those rents do pay for plenty of nice things, but it's really hard for me to ever understand Oracle or VMware as an "innovator", beyond their initial innovations (their flagship DB, x86 virtualization).
> Oracle has endured nearly 50 years. Sun did not endure.
IMHO it's perfectly fine for companies to live well, and then be sold. AFAIAC persistence is only proof of persistence. Sun created plenty of wealth/millionaires too. And, by Bryan's lights, it did so mostly ethically. That's a good life.
I think that companies like Oracle and Broadcom begin to resemble specialized private equity firms: they acquire innovative companies that have scaled to a level that they're familiar with. The acquirer then enforces "business discipline" and unlocks efficiencies (mainly this means leveraging the acquirer's existing connections with their customer base to cross-sell licenses, raising prices to the highest possible level their customers can sustain, and laying off/transferring redundant positions or positions not directly tied to revenue generation). This lasts 3-10 years until the market develops a lower-cost enterprise-ready alternative that starts to erode the captive customer base, but in that time these companies have collected enough rents to acquire another set of smaller companies and repeat the process.
> I think that companies like Oracle, SAP, and Broadcom begin to resemble specialized private equity firms
This is an entirely fair/accurate. I suppose what I am getting at is that these are just 2 different business models, and, the world can sustain a multitude of business models. There need not be only one (har har).
It's also fair to believe there is a moral dimension to one's own model which doesn't extract maximum value from the customer. Because IMHO "let's kick them in the dicks again" isn't an especially likable model, even if it is successful, and it's fair to avoid doing business with such people.
Imagine trying to sell your partners on doing business with Broadcom. If your core principle is "Broadcom needs to be around in 10 years", maybe the persistence/"kick them in the dicks" model is appealing, but otherwise, its fair for their competitors/Oxide to point out how awful dealing with a corporate sociopath might be.
Yeah, I agree with all of that. Extractive behemoths like Oracle couldn't exist without innovators like Oxide (well, not exactly Oxide since I doubt they'd sell to an Oracle, but you know what I mean).
There are plenty of customers that jump into deals with e.g. Oracle for a variety of reasons, and it's definitely worthwhile to spread the news far and wide about how difficult it is to work with these companies, doubly so if you're ideologically and economically competing with them.
I guess my point is that it's worthwhile to spend time understanding why this business model works in spite of all the shittiness, since the "hoping their poor treatment of customers will blow back on them" approach hasn't worked yet. I'm also fixating on the bad here, because I look at "both kind and nasty business models can succeed" and reflexively respond with "but why do the nasty ones succeed?"
At a practical level I think a thesis that "good" process isolation systems (aka, not hosted on Linux) build on years of development that unikernels will struggle to replace holds true.
At a conceptual level I really disagree with this piece, though:
> one cannot play up Linux kernel vulnerabilities as a silent menace while simultaneously dismissing hypervisor vulnerabilities as imaginary.
One can reasonably recognize Linux kernel vulnerabilities as extant and pervasive while acknowledging that hypervisors can be vulnerable. One can also realize that the surface area exposed by Linux is fundamentally much larger than that exposed by most hypervisors, and that the Linux `unshare` mechanism is insecure by default. It's kind of funny - the invocation of Linux really undermines this argument; there's no _reason_ a process / container isolation based system should be completely broken, but Linux _is_, and so it becomes a very weak opponent.
I really don't think I can agree with the debugging argument here at a conceptual level, either. Issues with debugging unikernels are caused by poor outside-in tooling, but with good outside-in tooling, a unikernel should be _easier_ to debug than a container or OS process, because the VM-owner / hypervisor will often already have a way to inspect the unikernel-machine's entire state from the outside, without additional struggle of trying to maintain, juggle, and restore multiple contexts within a running system. There is essentially an ISP/ICE debugging probe attached to the entire system end to end by default, in the form of the hypervisor.
For example, there is no reason a hosting hypervisor could not provide DTrace in a way which is completely transparent to the unikernel guest, and this would be much easier to implement than DTrace self-hosted in a running kernel!
If done properly, this way a uni-application basically becomes debugging-agnostic: it doesn't need cooperative tracepoints or self-modifying patches (and all of the state juggling that comes with that, think like Kprobe), because the hypervisor can do the tracing externally. The unikernel does not need to grow (in security surface area, debug-size, blast radius, etc.) to add more trace and debug capability.
Agree with your points and I think fundamentally the issue with unikernels at this point comes down to: nobody has really done it right yet.
By which I mean I see two variants:
1- exotic and interesting and constrained but probably not applicable for people in the form of e.g. MirageOS. not applicable because OCaml just isn't mainstream enough
2- Or other systems which allow much easier porting of existing systems by providing a libc and extended set of "porting" libraries which end up by recreating huge swathes of what the operating system is doing already anyways, in order to make the existing application just cross compile and "feel at home". But in reality probably always in an incomplete or odd way, and now you're using someone's hand crafted set of compatibility libraries instead of a battle tested operating system.
I just think we haven't seen the right system, yet, which would probably be some specific application development mostly from the ground up in the context of unikernel, not the other way around. Potentially a set of constrained and targeted Rust etc crates built from nostd up + some services. I kept looking for MirageOS for Rust and haven't seen, instead saw stuff more like 2.
In Qubes you use VMs to separate your banking environment from the one where you pull npm dependencies and the one where you open untrusted PDFs.
Networking also happens in its own VM, and you can have multiple VMs dedicated to networking.
Much lower memory footprint running mirage firewall, and an attack surface orders of magnitude smaller (compared to a VM running a Linux distribution purely for networking).
Cantrill is far smarter and accomplished than me, but this article feels a bit strawman and hit and run?
I think unikernels potentially have their place, but as he points, they remain mostly untried, so that's fair. We should analyze why that is.
On performance: I think the kernel makes many general assumptions that some specialized domains may want to short circuit entirely. In particular I am thinking how there's a whole body of research of database buffer pool management basically having elaborate work arounds for kernel virtual memory subsystme page management techniques, and I suspect there's wins there in unikernel world. Same likely goes for inference engines for LLMs.
The Linux kernel is a general purpose utility optimizing for the entire range of "normal things" people do with their Linux machines. It naturally has to make compromises that might impact individual domains.
That and startup times, big world of difference.
Is it going to help people better sling silly old web pages and whatever it is people do with computers conventionally? Yeah, I'd expect not.
On security, I don't think it's unreasonable or pure "security theatre" to go removing an attack surface entirely by simply not having it if you don't need it (no users, no passwords, no filesystem, whatever). I feel like he was a bit dismissive here? That is also the principle behind capability-passing security to some degree.
I would hate to see people close the door on a whole world of potentials based on this kind of summary dismissal. I think people should be encouraged to explore this domain, at least in terms of research.
If you software has no bugs then unikernels are a straight upgrade. If your software has bugs then the blast area for issues is now much larger. When was the last time you needed a kernel debugger for a misbehaving application?
> On performance: ... In particular I am thinking how there's a whole body of research of database buffer pool management
Why? The solution thus far has been to turn off what the kernel does, and, do those things in userspace, not move everything in the kernel? Where are these performance gains to be had?
> The Linux kernel is a general purpose utility optimizing for the entire range of "normal things" people do with their Linux machines.
Yeah, like logging and debugging. Perhaps you say: "Oh we just add that logging and debugging to the blob we run". Well isn't that now another thing that can take down the system, when before it was a separate process?
> That and startup times, big world of difference.
Perhaps in this very narrow instance, this is useful, but what is it useful for? Can't Linux or another OS be optimized for this use case without having to throw the baby out with the bathwater? Can't one snapshot a Firecracker VM and reach even faster startup times?
> On security, I don't think it's unreasonable or pure "security theatre" to go removing an attack surface entirely
Isn't perhaps the most serious problem removing any and all protection domains? Like between apps and the kernel and between the apps themselves?
I mean -- sure maybe remove the filesystem, but isn't no memory protection what makes it a unikernel? And, even then, a filesystem is usually a useful abstraction! When have I found myself wanting less filesystem? Usually, I want more -- like ZFS.
This is all just to say -- you're right -- there may be a use case for such systems, but no one has really adequately described what that actually is, and therefore this feels like systems autoeroticism.
> Why? The solution thus far has been to turn off what the kernel does, and, do those things in userspace, not move everything in the kernel? Where are these performance gains to be had?
There's all sorts of jankin' about trying to squeeze ounces of performance out of the kernel's page management, specifically for buffer pools.
Page management isn't really a thing we can do well "in user space". And the kernel has strong ideas about how this stuff works, which work very well in the general case. But a DB (or other system level things like garbage collectors, etc) are special cases, often with special needs.
LeanStore, Umbra, etc. do tricks with VMM overcommit and the like to fiddle around with this, and the above paper even proposes custom kernel modules for the purpose (There's a github repo associated, I'd have to go look).
And then, further, a DB goes and basically implements its own equivalent of a filesystem, managing its own storage. Often fighting with the OS about the semantics of fsync/durability, etc.
I don't think it's an unreasonable mental leap for people to start thinking: "I'm by necessity [cuz cloud] in a VM. Now I'm inside an OS in a VM, and the OS is sometimes getting in my way, and I'm doing things to get around the OS... Why?"
> Page management isn't really a thing we can do well "in user space".
But it is the thing most high performance OLTP DBMSs, most of us are aware of, do? I'm also not sure your cite is relevant here. Or it is at least niche. The comparison is made to LeanStore, which is AFAICT is not feature complete, and a research prototype?
Your cite does not describe a unikernel use case, but instead in kernel helper modules. Your cite is about leveraging in kernel virtual memory for the DB buffer cache, and thus one wonders how sophisticated the VM subsystem is in most unikernels? That is -- how is this argument for unikernels? Seems as though your cite is making the opposite argument -- for a more complex relationship between the DB and the kernel, not a pared down one.
> And then, further, a DB goes and basically implements its own equivalent of a filesystem, managing its own storage. Often fighting with the OS about the semantics of fsync/durability, etc.
The fights you're describing what have thus far been the problems of ceding control of the buffer cache to the kernel, via mmap, especially re: transactional safety.
If your argument is kernels may need a bottom up redesign to make ideas like this work, I suppose that makes sense. However, again, I'm not sure that makes unikernels more of an answer here than anywhere else, though.
> I don't think it's an unreasonable mental leap for people to start thinking: "I'm by necessity [cuz cloud] in a VM. Now I'm inside an OS in a VM, and the OS is sometimes getting in my way, and I'm doing things to get around the OS... Why?"
I think that's a fair thought to have, but the problem is how it actually works in practice. As in, less code seems really enticing, the problem is what abstractions are you throwing away. If the abstraction is less memory protection, maybe this is not a good tradeoff in practice.
I can see why he would make that argument. When you don't have any process isolation, a software fault means your entire stack is untrustworthy. The network driver, fs driver might be corrupted, so nothing you write to disk or send over the network can be trusted.
You also have to recreate your entire userspace and debugging tools to work in this environment, and testing or even running or debugging your software is also a headache.
damn… i am a big fan of bryan and i thought i was a big fan of unikernels… well, i still am, but all the points he makes are absolutely well-founded. i will say, in contraposition to the esteemed (and hilarious) mr. cantrill, that it is quite incredible to get to the end of an article about unikernels without seeing any mention of how the “warmup” time for a unikernel is subsecond whereas the warmup time for, say, containers is… let’s just call it longer than the warmup time for the water i am heating to make some pourover coffee after i finish my silly post. to dismiss this as a profound advantage is to definitely sell the idea more than a little short.
but at the same time i do think it is fair at this juncture to compare the tech to things like wasm, with which unikernels are much more of a direct competitor than containers. it is ironic because i can already hear in my head the hilarious tirade he would unleash about how horrific docker is in every way, debugging especially, but yet somehow this is worse for containers than for unikernels. my view at the present is that unikernels are fantastic for software you trust well enough to compile down to the studs, and the jury is still out on their efficacy beyond that. but holy fuck i seear to god i have spent more time fucking with docker than many professional programmers have spent learning their craft en toto, and i have nothing to show for it. it sucks every time, no matter what. my only gratitude for that experience revolves around (1) saving other peoples’ time on my team (same goes for git, but git is, indisputably, a “good” technology, all things considered, which redeems it entirely), and (2) it motivated me to learn about all the features that linux, systemd, et al. have (chroot jails, namespaces, etc.) in a way that surely exceeds my natural interest level.
> the “warmup” time for a unikernel is subsecond whereas the warmup time for, say, containers is… let’s just call it longer than the warmup time for the water i am heating to make some pourover coffee after i finish my silly post. to dismiss this as a profound advantage is to definitely sell the idea more than a little short.
I'm surprised to read that unikernels would start up much faster than containers. It seems like a unikernel needs to do more work (load kernel, and load app), in a more restricted way (hypervisor) than simply loading the app in a cgroup + namespace and letting it rip.
Are you sure this is an apples to apples comparison of similarly optimized images?
> to dismiss this as a profound advantage is to definitely sell the idea more than a little short.
Nah not really what he's saying. He's saying that if you throw out all the security affordances provided by page tables and virtual memory, it outweighs the "profound advantage" (which as he mentions, is arguable anyway since user/kernel context switch is a negligible cost in most modern systems).
You're selling a great deal in order to buy not much. It's a poor tradeoff.
Just to save people from wasting their time reading this drivel:
`
If this approach seems fringe, things get much further afield with language-specific unikernels like MirageOS that deeply embed a particular language runtime. On the one hand, allowing implementation only in a type-safe language allows for some of the acute reliability problems of unikernels to be circumvented. On the other hand, hope everything you need is in OCaml!
`
ToroKernel is written in freepascal.
All of the text before and after is completely irrelevant
Toro is just a library OS that allows you to build an application and deploy it as a VM without the OS. Toro acts as the OS. Different that other unikernels, Toro is not meant to be POSIX compliant. The idea is to provide an API that suits better the use-case, i.e., an app deployed as a VM. Toro can also run in baremetal although I dropped the support a few commits ago. I can roll back that support in case there is interest.
> So the prediction that incautious and unverified unsafe {} blocks would cause CVEs seems entirely accurate.
This is one/the first CVE caused by a mistake made using unsafe Rust. But it was revealed along with 159 new kernel CVEs found in C code.[0]
It may just be me, but it seems wildly myopic to draw conclusions about Rust, or even, unsafe Rust from one CVE. More CVEs will absolutely happen. But even true Rust haters have to recognize that tide of CVEs in kernel C code runs something like 19+ CVEs per day? What kind of case can you make that "incautious and unverified unsafe {} blocks" is worse than that?
> Github says 0.3% of the kernel code is Rust. But even normalized to lines of code, I think counting CVEs would not measure anything meaningful.
Your sense seems more than a little unrigorous. 1/160 = 0.00625. So, several orders of magnitude fewer CVEs per line of code.
And remember this also the first Rust kernel CVE, and any fair metric would count both any new C kernel code CVEs, as well as those which have already accrued against the same C code, if comparing raw lines of code.
But taking a one week snapshot and saying Rust doesn't compare favorably to C, when Rust CVEs are 1/160, and C CVEs are 159/160 is mostly nuts.
I'm more interested in the % of rust code that is marked unsafe. If you can write a kernel with 1% safe, that sounds pretty great. If the nature of dealing with hardware (AFAIK most of a kernel is device drivers) means something higher, maybe 10%, then maybe safety becomes difficult, especially because unsafety propagates in an unclear way since safe code becomes unsafe to some degree when it calls into it.
I'm also curious about the percentage of implicit unsafe code in C, given there are still compilers and linters checking something, just not at the level of lifetimes etc in Rust. But I guess this isn't easy to calculate.
I like rust for low level projects and see no need to pick C over it personally - but I think it's fair to question the real impact of language safety in a realm that largely has to be unsafe. There's no world where Rust is more unsafe than C though so it's all academic. I just wonder if there's been any analysis on this, in close to metal applications like a kernel.
> I'm more interested in the % of rust code that is marked unsafe.
I think you should less interested in % unsafe as what the unsafe is used to do, that is, it's likelihood to cause UB, etc. If it's unsafe to interface with C code, or unsafe to do a completely safe transmute, I'm not sure one should care.
> There's no world where Rust is more unsafe than C though so it's all academic
I think Rust is more unsafe than C due to supply chain issues in the Rust ecosystem, which have not fully materialized yet. Rust certainly has an advantage in terms of memory safety, but I do not believe it is nearly as big as people like to believe compared to a C project that actually cares about memory safety and applies modern tooling to address safety. There seems to be a lot of confirmation bias. I also believe Rust is much safer for average coders doing average projects by being much safer by default.
> I think Rust is more unsafe than C due to supply chain issues in the Rust ecosystem
This is such an incredibly cheap shot. First, the supply chain issues referenced have nothing to do with Rust, the language, itself. Second, Rust's build system, cargo, may have these issues, but cargo's web fetch features simply aren't used by the Linux kernel.
So -- we can have a debate about which is a better a world to live in, one with or without cargo, but it really has nothing to do with the Linux kernel security.
It would probably have to be normalized to something slightly different as lines of code necessary to a feature varies by language.. But even with the sad state of CVE quality, I would certainly prefer a language that deflects CVEs for a kernel that is both in places with no updates and in places with forced updates for relevant or irrelevant CVE.
The kernel policy for CVEs is any patch that is backported, no? So this is just the first Rust patch, post being non-experimental, that was backported?
> You don't generally need specific versions of GCC or Clang to build it I'm pretty sure.
You need a C11 compiler these days with loads of non-standard extensions. Note, for a very long time, one couldn't compile the Linux kernel with clang because it lacked this GCC specific behavior.
I'm not really sure you can turn around and say -- Oh, but now we feel differently about the C standard -- given how much is still non-standard. For instance, I don't believe Intel's C compiler will compile the kernel, etc.
> Every system under the Sun has a C compiler... My guess is that C will be around long after people will have moved on from Rust to another newfangled alternative.
This is still the worst possible argument for C. If C persists in places no one uses, then who cares?
> it always has been and always will be available everywhere
"Always has been" is pushing it. Half of C's history is written with outdated, proprietary compilers that died alongside their architecture. It's easy to take modern tech like LLVM for granted.
This might actually be a solved problem soonish, LLMs are unreasonably effective at writing compilers and C is designed to be easy to write a compiler for, which also helps. I don’t know if anyone tried, but there’s been related work posted here on HN recently: a revived Java compiler and the N64 decompilation project. Mashed together you can almost expect to be able to generate C compilers for obscure architectures on demand given just some docs and binary firmware dumps.
Which is neat! But if you care about performance, you're probably using Clang to compile for your esoteric architecture. And if your esoarch already supports LLVM... might as well write Rust instead.
I'm curious as to which other metric you'd use to define successful? If it actual usage, C still wins. Number of new lines pushed into production each year, or new project started, C is still high up the list, would be my guess.
Languages like Rust a probably more successful in terms of age vs. adoption speed. There's just a good number of platforms which aren't even supported, and where you have no other choice than C. Rust can't target most platforms, and it compiles on even less. Unless Linux want's to drop support for a good number of platforms, Rust adoption can only go so far.
> I'm curious as to which other metric you'd use to define successful? If it actual usage, C still wins.
Wait, wait, you can use C everywhere, but if absolute lines of code is the metric, people seem to move away from C as quickly as possible (read: don't use C) to higher level languages that clearly aren't as portable (Ruby, Python, Java, C#, C++, etc.)?
> Rust can't target most platforms, and it compiles on even less. Unless Linux want's to drop support for a good number of platforms, Rust adoption can only go so far.
Re: embedded, this is just terribly short sighted. Right now, yes, C obviously has the broadest possible support. But non-experimental Linux support starts today. GCC is still working on Rust support and rust_codegen_gcc is making in roads. And one can still build new LLVM targets. I'd be interested to hear what non-legacy platforms actually aren't supported right now.
But the real issue is that C has no adoption curve. It only has ground to lose. Meanwhile Rust is, relatively, a delight to use, and offers important benefits. As I said elsewhere: "Unless I am wrong, there is still lots of COBOL we wish wasn't COBOL? And that reality doesn't sound like a celebration of COBOL?" The future looks bright for Rust, if you believe as I do, we are beginning to demand much more from our embedded parts.
Now, will Rust ever dominate embedded? Will it be used absolutely everywhere? Perhaps not. Does that matter? Not AFAIAC. Winning by my lights is -- using this nice thing rather than this not so nice thing. It's doing interesting things where one couldn't previously. It's a few more delighted developers.
I'd much rather people were delighted/excited about a tech/a language/a business/a vibrant community, than, whatever it is, simply persisted. Because "simply persisted", like your comment above, is only a measure of where we are right now.
Another way to think of this is -- Perl is also everywhere. Shipped with every Linux distro and MacOS, probably to be found somewhere deep inside Windows too. Now, is Perl, right now, a healthy community or does it simply persist? Is its persistence a source of happiness or dread?
Why do you think people are delighted/excited about Rust? Some people are, but I’m sure plenty of people are about C and Perl is well. Some people isn’t most people and certainly not all people.
> Some people isn’t most people and certainly not all people.
I never said Rust was the universal language, that is pleasing to all people. I was very careful to frame Rust's success in existential, not absolute or universal, terms:
>> Winning by my lights is -- using this nice thing rather than this not so nice thing. It's doing interesting things where one couldn't previously. It's a few more delighted developers.
I am not saying these technologies (C and Perl) should go away. I am saying -- I am very pleased there is no longer a monoculture. For me, winning is having alternatives to C in embedded and kernel development, and Perl for scripting, especially as these communities, right now, seem less vibrant, and less adaptable to change.
>> Yes, you can use it everywhere. Is that what you consider a success?
> ... yes?
Then perhaps you and I define success differently? As I've said in other comments above, C persisting or really standing still is not what I would think of as a winning and vibrant community. And the moving embedded and kernel development from what was previously a monoculture to something more diverse could be a big win for developers. My hope is that competition from Rust makes using C better/easier/more productive, but I have my doubts as to whether it will move C to make changes.
Sometimes it's nice to know that something will run and compile reliably far into the future. That's a nice thing to have, and wide support for the language and its relatively unchanging nature make it reliable.
> Sometimes it's nice to know that something will run and compile reliably far into the future.
I'm not sure why you think this is a problem that Rust has? Perhaps you mean something different but the Rust project compiles available code on crate.io upon the release of a new version.[0] C compilers may imagine their new compiler code doesn't break old software, but Rust takes that extra step, so we know it won't.
Now, the Rust kernel is currently using beta and nightly features which are on track for inclusion in the stable Rust compiler. So, yes, right now compilation is tied to a specific kernel version, and may need to be updated if a feature changes. However, any C compiler used to compile the Linux kernel uses non-standard GCC extensions only recently adopted by clang. Imagine if the C standards committee chose to change the syntax/sematics of a non-standard extension. Do you not imagine the non-standard extension would also be similarly deprecated?
The issue seems to be Rust is telling you what is non-standard, and you're yelling "Look it's non-standard!". But consider that the kernel in practice is using lots of non-standard features, and should the C standard simply adopt this non-standard behavior that means likely having to changes lots of code.
You almost certainly have a bunch of devices containing a microcontroller that runs an architecture not targeted by LLVM. The embedded space is still incredibly fragmented.
That said, only a handful of those architectures are actually so weird that they would be hard to write a LLVM backend for. I understand why the project hasn’t established a stable backend plugin API, but it would help support these ancillary architectures that nobody wants to have to actively maintain as part of the LLVM project. Right now, you usually need to use a fork of the whole LLVM project when using experimental backends.
> You almost certainly have a bunch of devices containing a microcontroller that runs an architecture not targeted by LLVM.
This is exactly what I'm saying. Do you think HW drives SW or the other way around? When Rust is in the Linux kernel, my guess is it will be very hard to find new HW worth using, which doesn't have some Rust support.
In embedded, HW drives SW much more than the other way around. Most microcontrollers are not capable of running a Linux kernel as it is, even with NoMMU. The ones that are capable of this are overwhelmingly ARM or RISC-V these days. There’s not a long list of architectures supported by the modern Linux kernel but not LLVM/Rust.
Almost none of them, but that doesn’t make them “places no one uses”, considering everyone who can see these words is looking at a device with a bunch of C firmware running on microcontrollers all over the place.
You're putting words in my mouth there - that's not my point.
My point is that if they're not running linux, and the ones that are aren't running a modern kernel, then we shouldn't hold back development of modern platforms for ones that refuse to keep up.
> I currently have no reason to believe C won't outlive it, by a lot.
My reaction is kind of: "So what?" I really don't care about the relative lives of languages and don't really understand why anyone would. Unless I am wrong, there is still lots of COBOL we wish wasn't COBOL? And that reality doesn't sound like a celebration of COBOL?
IMHO it would be completely amazing if magically something 10x better than Rust came along tomorrow, and I'd bet most Rust people would agree. Death should be welcomed after a well lived life.
To me, the more interesting question is -- what if efforts like c2rust, Eurydice, TRACTOR and/or LLMs make translations more automatic and idiomatic? Maybe C will exist, but no one will be "writing" C in 20 years? Perhaps C persists like the COBOL zombie? Perhaps this zombification is a fate worse than death? Perhaps C becomes like Latin. Something students loath and are completely bored with, but are forced to learn simply as the ancient interface language for the next millennia.
Is that winning? I'd much rather people were excited about tech/a language/a business/vibrant community, than, whatever it is, simply persisted, and sometimes I wish certain C people could see that.
I plan to be writing C for the next decades even for new projects, because I think it is a great language, and I appreciate its simplicity, fast compilation times, stability, and portability.
I am happy if people are excited about Rust, but I do not like it too much myself. Although I acknowledge that it contains good ideas, it also has many aspects I find problematic and which why I do not think we should all switch to it as a replacement for C.
This is fantastic, if you're still excited about C. However, the question I am asking also has to do with the social relational aspects of programming. Here, significantly, when we are talking about the lives of programming languages/decades, I think we should ask whether other people, and, importantly, industry, will be equally jazzed about yours and others C code, in the future. Are people still going to be happy to deal with all of C's problems when, perhaps, alternatives exist, and if, say, memory safety becomes cheaper, etc.?
Mostly the advantages a listed for C: stability, portability, simplicity, fast compilation times could all in reverse also be considered disadvantages of Rust. I am also not a fan of monomorphization, not of static linking, and not of having many dependencies out of a large uncurated pile of many projects. I also think Rust is not pragmatic enough. Overall though, my main issue is complexity of the language which I think goes in the wrong direction - this may be ok if the alternative is C++, but if you prefer C to C++ then Rust is equally unappealing. At the same time I think the advantages of Rust are exaggerated. I still agree memory safety is important, I just think Rust is not an ideal approach.
With LLMs becoming so good at coding, "just never make any mistake" is also becoming easier. I usually write my own code because I don't like the coding style of LLMs, but on more than one occasion now they have proven very useful as code reviewers and bug hunters.
We were talking about Rust issues. But yes, C should have a proper string type. But as long as it does not have a better standardized string type, it is possible to define your own (or use a library).
C will end when our entire technological civilization collapses and has to start over from scratch and even then after a century of progress they will invent C again.
C isn't a bad language per-se, it just doesn't have an opinion on most things and I think that's exactly the reason why it survived many higher level (and more opinionated) languages that also already existed when C was created.
This is exactly right. The trend is towards comprehensive programming frameworks for convenient programming with batteries included. I hope this trend dies and we focus more on integration into the overall ecosystem again.
I haven't ever had to do anything serious in C but it's hard to imagine getting it 100% right.
A while back I wrote some C code to do the "short-bread" problem (it's a bit of a tradition at work to give it to people as their first task, though in Python it's a lot easier). Implementing a deque using all of the modern guard rails and a single file unit test framework still took me a lot of attempts.
Nowadays, not so much. Computers are multiple orders of magnitude faster, have multiple orders of magnitude more memory and storage and do things multiple orders of magnitude more complex than they used to. Portable assembly still has its uses obviously, but safer/easier/faster alternatives exist in all its niches.
C is much closer to any other high level language than it is to assembly. 'Portable assembly' might have been true with trivial C compilers of the 70s and 80s, but not with compilers like gcc or clang.
> COBOL has enough business money around to get new tools and ISO standards[0], so it is unlikley to think otherwise regarding C.
I don't think you understand my point. I am explicitly saying "C will definitely survive (like COBOL)". I am asking is that the kind of life people want for C?
Ideally we would have moved on into some Assembly glue + compiled managed high level languages by now, like Xerox PARC when then moved away from BCPL into Smalltalk, Interlisp-D, Mesa and Mesa/Cedar, but some folks and industry standards cannot let go of C, and those have to contend with that kind of life for C, exactly.
C has never been a particularly good language, and is so good that finally (with tons of pushbacks!) there is an alternative that make the case so strong that is at least considered the possibility that will come the very happy day where C will be our past.
The only, true, real blocker is that C is the ABI. But if we consider the possibility that C can AND should be the past, then the C Abi can finally adds sophisticated things like Strings and such, and maybe dreaming, algebraic types (ie: the C will be improved with the required features so it can evolve the ABI, but not because will be a good language for write it (manually) on it).
And to reiterate: C should finally be a real assembly language, something we not need to worry about.
> And to reiterate: C should finally be a real assembly language, something we not need to worry about.
Assembly is used quite a lot and if you're a programmer Assembly is very valuable to know _at least_ how to understand it.
I disagree, also, that C should go away. Saying it was never a good language is a bit harsh. It's a great language. One that industries are built on. I'd rather read/write C code than, say, Rust.
Edit:
There are, of course, languages coming up that can absolutely compete with C. Zig could be one when it's mature, for instance.
Sane, easily readable syntax and expressive semantics. Easy to learn. Very scalable. Suitability, by design, for low level systems programming, including microcontrollers. Suitability, by design, for
large, complex real-time applications. Easy to interface with C and other languages. Available as part of GCC. Stable and ongoing language evolution.
Honestly I'd be a bit disappointed if something better came along tomorrow. Just as we as an industry spent all this effort moving to Rust something better comes along? Lame. Obviously I want better languages to come out, but I'd either want a bit of warning or a slower pace so we as an industry don't totally "waste" tons of time on transitioning between short-lived languages. Thankfully languages need about 10 years to mature from 0.1 to production readiness, and industry happily ignores marginally (and moderately) better languages than what they're using, so this is not a realistic issue.
If all Rust accomplishes is ushering some other better project, it would have been worth it.
I think it would take a while for that to happen, purely due to momentum' the same thing that makes some people think that Rust isn't being used will affect any younger language just as much, if not more.
I think that there's an easier language than Rust struggling to come out of it, but if Rust had been that easier language with different compromises, I doubt it would have gained critical mass that allowed it to get where it is today. Being fast and safe meant it carved a niche in a "free square" that drove it to have a clear differentiator that allowed it to gain an initial audience. I also suspect that it is easier toale a language fast and then evolve it to make it easier to use, than it is to make it easy to use first and then make it fast.
Note I ignored the 10x part. I'd find it a bit lame if a language came out that's 1.1x better than Rust because we're now in the awkward position of having rewritten lots of stuff in the second best language. However, should a 10x language come out you'll just have to swallow all that bitterness and start over because 10x is 10x.
Obviously a 1.1x language will come out - we don't just jump directly to 10x - and that's fine, fantastic even, but a little bit annoying when you're a language enthusiast and you've personally spent lots of time advocating for the now next-best language.
> Honestly I'd be a bit disappointed if something better came along tomorrow.
You'd be disappointed if something 10x better came along tomorrow? I suppose you would you also be disappointed if magically we had economical fusion power, because you own utility stocks? Or we invented 10x better new car, because you already own an old car?
Of course the world wouldn't immediately move to one thing or the other, etc., and we'd still have a 10x better thing?
> Obviously I want better languages to come out, but I'd either want a bit of warning or a slower pace
The purpose of this thought experiment is to say -- it's perfectly fine for things to live and die, if they must. We've had a second Cambrian period for PLs. It's perfectly alright if some don't live forever, including Rust, which I really like.
In my thought experiment, Rust and C could also accept this new paradigm, and adapt, and perhaps become 10x better themselves. Though this is something heretofore C/C++ haven't done very well. IMHO new things don't preclude old things, and there mustn't be only one winner.
> Thankfully languages need about 10 years to mature from 0.1 to production readiness, and industry happily ignores marginally (and moderately) better languages
Which my thought experiment did as well? Read: This is a 10x improvement!
Oops, skipped the 10x part. If it's really 10x better that would indeed be amazing. That's basically the leap from C to Rust in domains that C is not good at.
A scrub literally reads the object from disk. And, for each block, the checksums are read up the tree. The object is therefore guaranteed to be correct and consistent at least re: the tree of blocks written.
> Unfortunately it's possible for ZFS bugs and issues to give you filesystem objects that have problems
Can you give a more concrete example of what you mean? It sounds like you have some experience with ZFS, but "ZFS doesn't have an fsck" is also some truly ancient FUD, so you will forgive my skepticism.
I'm willing to believe that you request an object and ZFS cannot return that object because of ... a checksum error or a read error in a single disk configuration, but what I have never seen is a scrub that indicates everything is fine, and then reads which don't return an object (because scrubs are just reads themselves?).
Now, are things like pool metadata corruption possible in ZFS? Yes, certainly. I'm just not sure fsck would or could help you out of the same jam if you were using XFS or ext4. AFAIK fsck may repair inconsistencies but I'm not sure it can repair metadata any better than ZFS can?