Some of this exists already in pockets (Common Crawl, The Pile, RedPajama are all volunteer/open efforts). I suppose there's no equivalent of the "edit this page and see the impact" like with have with Wikipedia. Contributing to an open dataset has no feedback loop if the training infrastructure that would consume it is closed... seems like a feedback problem.
Agree that this makes it unlikely we see frontier training data OS'd but this is a separate problem from software and infrastructure transparency, which has none of those constraints. Training stack, the parallelism decisions, documented failure modes are engineering knowledge and there's no principled reason it doesn't ship.
The framing here is undersold in the broader discourse: "open weights" is a ruse for reproducibility. What you have is closer to a compiled binary than source code. You can run it, you can diff it against other binaries, but you cannot, in any meaningful sense, reproduce or extend it from first principles.
This matters because OSS truly depends on the reproducibility claim. "Open weights" borrows the legitimacy of open source (the assumption that scrutiny is possible, that no single actor has a moat, that iteration is democratised). Truly democratised iteration would crack open the training stack and let you generate intelligence from scratch.
But how useful is source code if it takes millions of dollars to compile? At that point, if you do need to make changes, it probably makes more sense to edit the precompiled binary. Even the original developers are doing binary edits in most cases.
I agree that open weight models should not be considered open source, but I also think the entire definition breaks down under the economics of LLMs.
There are lots of reasons to read through source code you never edit or recompile: security audits, interoperability, learning from their techniques, etc. And I think many of those same ideas apply to seeing the training data of a LLM. It will help you understand quickly (without as much experimentation) what it's likely to be good at, where its biases may be, where some kind of supplement (transfer learning? RAG? whatever) might be needed. And the why.
If you are unable to run the multimillion training, then any kind of security audit of the training code is absolutely meaningless, because you have no way to verify that the weights were actually produced by this code.
Also, the analogy with source code/binary code fails really fast, considering that model training process is non-deterministic, so even if are able to run the training, then you get different weights than those that were released by the model developers, then... then what?
I probably shouldn't have led with that example because yeah, reproducible (and cheap) builds would be best for security audits. But I wouldn't say it's absolutely meaningless. At least it can guide your experimentation, and if results start differing radically from what you'd expect from the training data, that raises interesting questions.
If you're going through the effort to be open source you can probably set up fixed batch sizes and deterministic combination of batches without too much more effort. At least I hope it's not super hard.
I have zero actual experience in training models, but in general, when parallelizing work: there can be fundamental nondeterminism (e.g., some race conditions) that is tolerated, whose recording/reproduction can be prohibitive performance-wise.
Agree, this feels like a distinction that needs formalising...
Passive transparency: training data, technical report that tells you what the model learned and why it behaves the way it does. Useful for auditing, AI safety, interoperability.
Active transparency: being able to actually reproduce and augment the model. For that you need the training stack, curriculum, loss weighting decisions, hyperparameter search logs, synthetic data pipeline, RLHF/RLAIF methodology, reward model architecture, what behaviours were targeted and how success was measured, unpublished evals, known failure modes. The list goes on!
I'd also add training checkpoints to the list for active transparency. I think the Olmo models do a decent job, but it would be cool to see it for bigger models and for ones that are closer to state-of-the-art in terms of both architecture and algorithms.
Security audits, etc, are possible because binary code closely implements what the source code says.
In this case, you have no idea what the weights are going to "do", from looking at the source materials --- the training data and algorithm --- without running the training on the data.
Compute costs are falling fast, training is getting cheaper. GPT-2 costs pocket change to train, and now it costs pocket train to tune >1T parameter models. If it was transparent what costs went into the weights, they could be commodified and stripped of bloat. Instead the hidden cost is building the infrastructure that was never tested at scale by anyone other than the original developers who shipped no documentation of where it fails. Unlike compute, this hidden cost doesn't commodify on its own.
yeah, the costs are definitely a factor and prohibitive in completely replicating an open source model. Still, there's a lot of useful things that can be done cheaply, including fine tuning, interpretability work, and other deeper investigations into the model that can't happen without the infrastructure.
The training methods are largely published in their open research papers - though arguably some open weight companies are less open with the exact details.
Realistically a model will never be "compiled" 1:1. Copyrighted data is almost certainly used and even _if_ one could somehow download the petabytes of training data - it's quite likely the model would come out differently.
The article seems to be talking more about the difficulties of fine tuning models though - a setup problem that likely exists in all research, and many larger OSS projects that get more complicated.
> "Open weights" borrows the legitimacy of open source
I don't really see how open-weights models need to borrow any legitimacy. They are valuable artifacts being given away that can be used, tested and repurposed forever. Fully open models like the OLMo series and Nvidia's Nemotron are much more valuable in some contexts, but they haven't quite cracked the level of performance that the best open-weights models are hitting. And I think that's why most startups are reaching for Chinese base LLMs when they want to tune custom models: the performance is better and they were never going to bother with pretraining anyway.
Personally, I love this theory. The thought of natural assembly and selection at the level of Black Holes is alluring. Not sure what The Black Mirror Hypothesis (https://curtjaimungal.substack.com/p/when-you-fall-into-a-bl...) would have to say about this, though.
I've been calling out the similarity of works done by Barbour, Turok, Farnes & Petit for a long time, and the last developments by Turok's team vindicate this intuition. It is now very close to Jean-Pierre Petit's Janus model. Curt Jaimungal announced he'd interview him soon.
They were one of the earliest to adopt bitcoin and monero payments--if they didn't convert all those payments immediately to cash, they're probably sitting pretty right now.
They also have a partnership with Tailscale that can't be undersold.
I'm not sure how much it adds to their bottom line for each sale, but my corp was using the Mullvad VPN addition to tailscale to do global testing by our developers.
IE; "is something blocked, do we detect GEOIP properly" etc;
> there's definitely been a lottery win or a series A
We have neither won the lottery nor taken on outside investment. We've been growing for years, and we've reached a point where we can afford campaigns like this. It is an interesting experiment by our marketing team. Still, I think people on HN overestimate the cost of campaigns like this.
The National Library of Scotland has an impressive web tool that allows you to overlay detailed historical Ordnance Survey (OS) maps (dating back to 1841 for this instance) onto modern satellite imagery. The interface lets you adjust transparency and blend between past and present landscapes.
reply