My pet theory is that while the pretraining -> RL pipeline achieves very impressive results, it does not reward clarity of thought or elegance. It's not obvious whether it even should for most tasks, but it does grind on me as a human who needs elegance in order to keep everything under control. You give astra/codex many tasks, it retires them all more efficiently than I could by hand. But you look under the hood and every bugfix is another codepath, it just hammers away at things with admirable persistence and vigor until the tests pass. Similarly in discussions and docs, I've noticed many LLMs like to "beat around the bush."
it does not reward clarity of thought or elegance.
Well duh. RL can only train behaviors that can be defined. Clarity and elegance are damn subjective.
Also, I doubt we'll ever get AI to understand what clarity is to a human. They have such enormous contexts that what's clear to them is not clear to us.
Subjective doesn't mean you can't define it. RL works well for human preference and any other subjective goal as long as it's defined. In fact Claude used to be trained for human likeness, reading between the lines, character, languages, and understanding the intent. Claude 3 was outstanding in that. It was their entire marketing shtick. Then they neglected it starting with Sonnet 3.5, ignored it in favor of code since Claude 4.0, and new models completely lost their ability to write and understand humans. Code pays better.
Okay any argument about why space is uniquely challenging is going to revolve around physics. Sure it's not literally physically impossible, but we need to explain to people why this is different from shipping the GPUs to Ohio.
If you want math then https://andrewmccalip.com/space-datacenters exists. The numbers are grim for orbital DC. Even if you drag the launch cost slider all the way to $1/kg (by the way this is literally sci-fi, per ChatGPT air freight of semiconductors from Taiwan to Ohio costs $9/kg and ocean/train freight costs a bit under $1/kg for a reasonable shipment so good luck with $1/kg to LEO this century) it is still more than twice as expensive as terrestrial DCs.
But that calculator shows most of the cost is in the satellite. At $8 per watt (or about $2 million per satellite) the cost of orbital compute matches terrestrial.
That sounds absolutely possible. But in any event, we're now arguing a different thing.
The only thing I'm arguing is that cooling is a solved problem. I don't know if SpaceX will ever get the price down so that it is economical. But I'm convinced that there is no universe in which they hit their foreheads and say, "Oh no, we forgot about cooling!"
Note that this calculator is actually quite optimistic for orbital wrt. many things including cooling and effect on launch, as:
> No additional mass for liquid cooling loop infrastructure; likely needed but not included
> Thermal: only solar array area used as radiator; no dedicated radiator mass assumed
In hardware and mfg. solvable vs. solved is a big difference. And I too believe that SpaceX's engineers know about radiator panels. But the more cynical interpretation is that whatever the SpaceX engineers think about the technical merits, they are not being asked for that. They are just being asked for a pretext that justifies the xAI acquisition. Elon is also discussing lunar satellite factories that launch the satellites via railgun. Now, is this physically impossible? No, that isn't physically impossible either and I will seriously defend the physical possibility of this. It's not going to happen though.
And you could spend all the engineering costs on building some seriously efficient terrestrial DCs, but somehow all these analyses start with "assume that launch and satellite technology advances manyfold and terrestrial DCs stagnate or become less efficient, then if you squint the two numbers get kinda close."
Certainly launch has to advance a lot for this to make sense. SpaceX is betting big on Starship getting close to $100/kg. But does satellite tech have to advance much at all to make this work? SpaceX has already successfully deployed one megaconstellation into orbit and it's extremely profitable. Isn't this just another Starlink with bigger solar panels and radiators?
Yes it has to advance a lot. In a DC all your supporting equipment has returns to scale and can be repaired if it breaks. If a GPU breaks, a sysadmin walks over to the offending rack and swaps the card. In space all that equipment serves just a few cards (this is more like orbital server racks) and it has to work in space (so instead of using an ~infinite heat sink like the Earth, you use radiators etc). We still don't exactly know what effect radiation in LEO would have on stock GPUs - IIRC the experiments to determine this started after Elon went all in on orbital DCs. If anything breaks, you have a flying brick.
This is why that calculator, even under extremely optimistic assumptions for orbital, and even if you assume launch is zero, still cannot make it competitive with terrestrial.
Doesn't Starlink already handle all that though, just with networking racks instead of GPU racks? What's so unique about GPUs compared to other computer equipment, besides the power budget?
Commercial GPUs already fail a lot and that's before you put them in LEO. Starlink is doing something that can't be accomplished anywhere else than space. If you could somehow have ground stations that can give wireless internet to every corner of the Earth you would much rather do that. Again, this is all physically possible, but the question is whether it's wise, whether it's saving money or if it's just a giant Rube Goldberg machine. If you assume Starlink satellite $/W then even if launch is zero, it costs twice as much as terrestrial using assumptions charitable to orbital.
Fair point about the failure rates. I wonder what causes that, given that there are no moving parts... if it's temperature fluctuations, maybe you could just run the GPU at 100% all the time to avoid that, given that the marginal cost of power for a satellite already in space is $0?
> If you assume Starlink satellite $/W
Why would you assume that, given that high power output is not a design goal of Starlink?
The calculator I've cited elsewhere uses https://chatgpt.com/share/69391474-4b24-8005-bb93-ebd4340c65.... The various starlink versions have approximately the same cost per watt across very different power levels. Much of the cost is going to be ~proportional to power - solar panels, heat rejection, power electronics, etc. Listen, if they could make it cheaper to deliver the same amount of power they would. Starlink is approximately all of SpaceX's business.
ETA: power is absolutely important for Starlink. SNR is very important for shannon capacity and satellite systems are often limited in this respect.
So ChatGPT decided to assume a constant $800/kg to estimate cost, and you're using that as evidence that cost/W doesn't decrease with scale?
Anyway, my point is that Starlink is not primarily designed to harvest as much solar power as possible the way data center sats are. I'm sure they're not making it inefficient on purpose, but what you're suggesting is like using the cost/W of a solar powered traffic camera to estimate the cost/W of a solar farm. Yes, they both use solar power, but they have entirely different design goals.
It's reasonable to 0th order to assume a ball of heat management and phased array antennae has a roughly constant cost per kg, yes. And as I said starlink absolutely wants to use as much power as they can, as communications from space are heavily limited by power.
FWIW, SpaceX is estimating their AI sats will have over double the W/kg of what that ChatGPT summary is estimating for Starlink. https://www.spacex.com/spacexai/starmind Cost/kg is not disclosed, but I think it would be a mistake to assume there were no similar improvements there.
Okay so even assuming all of SpaceX's numbers, which is extremely optimistic, even if we're an angel's advocate and assume that they spend all this money on engineering and optimize their launch costs and their satellite costs and they don't hit any hiccups, and DCs on Earth stay pretty much the same and don't get any added efficiencies. Even then over every time period orbital is still more expensive. And it gets particularly punishing the longer you extend the analysis period as the satellites need to get replaced while the DCs don't spontaneously combust all their infra.
For two different indivisible charges, this is a bit slippery but I think a=1 by definition. How do we measure charge, practically speaking? By how much force is measured between it and a reference charge. So we take a=1 as a convention. But no experiment can disprove a=2 for indivisible charges as we would simply obtain charge through new units. For assemblies of charges the forces must add linearly due to conservation of momentum, so there we know a=1.
I tested both myself and a weak bot against Astra xhigh, https://lichess.org/study/27lCQqDa. It's still pretty bad at chess, though it takes longer to devolve into illegal moves.
It looks like it played a fully legal game of chess with one exception, it said "rxd1+" (Rook takes D1 with check) instead of "rd1+" (Rook to D1 with check) on move 29.
I would say this did a really good job of playing chess. It moved the pieces consistently and traded pieces when required.
This is worlds away from the frontier ~1 year ago where models would hallucinate pieces into existence.
You can still see undercurrents of its old self, once I pointed out the illegal notation it hallucinated prior illegal moves. But I agree, it's leagues apart from prior iterations. It also knew thematic moves in the opening. But whenever it needs to play concretely rather than "I know so-and-so is a good move in these types of positions" it crumbles.
Would you tell a human that just tried doing an illegal move that they did "a really good job of playing chess"? The probability for such mistakes is greatly reduced but still far from negligible, which proves the point that guardrails are needed.
So you weren't giving it an updated board state after every move? If you want to compare apples to apples, it should give an updated board state for each move, or you should play blindfolded.
Blindfolded flex by OP aside (I can barely play when seeing the board), considering reasoning traces and their nature, if we want to be fair, a person would have to get the moves, but be allowed to write them down or draw up a board in their notepad. My working memory can barely handle five chunks, a models reasoning tokens are masses of written text in comparison.
An LLM has been trained to do everything it does blindfolded, "only" using perfect recall of everything in it's hundreds of thousands of steps of context, and hundreds of layers of KV cache. It's a computer - it has a massive advantage over a human.
The fairest apples-to-apples comparison of an LLM whose training data included chess games would be a trained human such as Magnus Carlson, who can quite happily play a dozen or more simultaneous blindfold chess games.
> though it takes longer to devolve into illegal moves
Is this because the context is being saturated? How did you set it up?
Was the prompt something like "Here's the state of the board, you're white, your move, what do you do?" and then starting fresh each time? Or did it include the whole history of moves and board states and previous thinking tokens and so on? No judgment, just trying to add this data point (thanks for sharing!) to my mental model and understanding.
I'd be curious how it would work if it started fresh each time. My guess is it would never make an illegal move, although it may not actually play all that well.
Any claim that AI will, or could, destroy humanity reduces to a claim that any sufficiently intelligent being - even a human - could destroy humanity. I find that much of the x-risk thought relies on religious thinking. Take for example: https://x.com/paulg/status/1660404244174782464.
If this line of thinking is taken too literally, we can never falsify it. Any specific hypothesis - nukes, bioweapons, spontaneously convincing us that life isn't worth living - can be deflected with the objection that if we can anticipate it and prevent it, it is not the route for a true ASI extinction event.
I'm not an AI doomer but, it does not reduce to that claim. The threat is not just one smart being. The threat is beings that and (1) multiply themsevles instantly, unlike humans that take 15+ years (2) share knowledge instantly "I know kung-fu" matrix style. Humans can share knowledge but they can not absorb it like an AI can/could/will. Human armies have conquered other humans. An army with infinite soldiers will win against one with finite soldiers
Yes, you'll come up with all kinds of objections like AI doesn't have presence in the physical world, etc... That's fine. I'm not arguing my example is perfect, I'm only arguing your characterization of one smart being is not the threat being considered.
They don't multiply themselves instantly, and this is exactly the kind of religious, magical thinking to which I refer. Any thinking machine is an amalgam of hardware and software. The software changes with exceptional flexibility, yes. But if you attempt to create an "infinite" number of these you will quickly run into roadblocks.
No one serious is really arguing for a scenario where the machines rise up a la Planet of the Apes. The dangers people are really examining are scenarios of either one exceptionally clever innovation: a designer virus or hacking NORAD; or cleverly amassing economic / political influence over time akin to an exceptionally clever tech magnate. And these roles could adequately be filled by either ASI or a Bond villain.
We must have different definitions. Millions of them are created daily. Every time an agent is spawned = creation of an agent and millions of people are creating multiple agents daily.
First don't get fixated on the "entire" humanity part. Even 10% should be unacceptable period.
Second you can't compare it to just one person's capability.
And it's never a doubt that really smart and insane people absolutely could kill a lot of humans considering modern technology. The reason they don't is because our society was historically built so that smart people don't want to or get stopped. For example ethics, religion, mental hospitals, self preservation instinct, police etc.
If anyone believes a sufficiently smart consortium of humans could exterminate 10% of humanity, that alone should be fixed by identifying the pathways through which it could be done. The existence of AI is not needed to be concerned about that possibility.
And dealing with those pathways individually would be far more productive than attempting to control the proliferation of algorithms that think, which in the long term is probably impossible. It also leaves us on far firmer scientific footing. An individual pathway - bioweapons, nuclear weapons, etc. - is far easier to reason about and accept/reject a notion of feasibility. Treating an ASI as a machine god and asserting futility in face of that god is not going to get anything done.
Part of the point in all this is that we can't predict what an ASI would do. All we can say for certain is that we would not be able to control it, because that follows a simple logic that between two entities with differing goals, the more intelligent entity is more likely to see their goals realized. It's interesting to me that you seem adamant that we identify and fix the pathways through which smart humans could exterminate 10% of humanity, but fail to see that preventing ASI is doing exactly that.
Not to mention, it relies on a ton of completely undefined concepts.
No one can actually tell you what ASI is or entails because it isn't a legitimate, operational concept. It is a fairy tale.
We can't even define "alignment". As people have finally started pointing out, humanity has never had a collective agreement on what values it should uphold or what ultimate goods are. Your alignment is not my alignment.
AI is not even autonomous. Every system we have today has to be initiated by a human actor. "AI" wouldn't create catastrophic bio weapons, it would help humans create them. The humans are the source of the intent.
Everyone has just completely given in to empty language and marketing nonsense. Honestly it seems like been the people at the labs are drinking their own kool aid and are themselves deeply confused about what they are even building at this point. It is a stateless statistics function running on a bunch of data centers. We aren't even close to an embodied, conscious synthetic being. It doesn't even have state, which is like prerequisite number one, nor is it plastic.
I'm making an online guidance solver for KSA (https://ahwoo.com/app/100000/kitten-space-agency). It will allow you to control a KSA rocket from the command line for a variety of maneuvers (launch, rendezvous, etc). It uses successive convexification to turn the full nonlinear problem into a sequence of convex problems with linear constraints and quadratic objective.
It has proven surprisingly resistant to AI so far - Astra can make a very quick prototype but upon review it had many defects. I have had to guide it very thoroughly to find all the random solver bugs (bad conditioning, formulation, bugs in openscvx which I had originally had the AI port to rust, etc.) preventing well-behaved solves. But I think (hope?) I have turned the corner on these.
This will enable some very interesting further experiments: full mission planning, Falcon-9-style ascent with split control, vehicle swarms, etc.
Both labs even explicitly promise the customer owns the outputs. It feels like they want to have their cake (ensure enterprises don't get spooked away from using as many LLMs as possible) while eating it too (still arguing some level of control over the outputs).
> Ownership of content. As between you and OpenAI, and to the extent permitted by applicable law, you (a) retain your ownership rights in Input and (b) own the Output. We hereby assign to you all our right, title, and interest, if any, in and to Output.
> As between the parties and to the extent permitted by applicable law, Anthropic agrees that Customer (a) retains all rights to its Inputs, and (b) owns its Outputs. Anthropic disclaims any rights it receives to the Customer Content under these Terms. Subject to Customer’s compliance with these Terms, Anthropic hereby assigns to Customer its right, title and interest (if any) in and to Outputs.
> Both labs even explicitly promise the customer owns the outputs.
> to the extent permitted by applicable law, you (a) retain your ownership rights in Input and (b) own the Output
If the argument is that the model itself is under copyright protection then "as permitted by applicable law" would be doing some heavy lifting. Assuming that were true, given that locally-run LLMs exist, what would be illegal: the distillation itself or the provision of service of the distilled model?
I don't really have a strong opinion either way on Flock but this is a strange objection. YC is a well-known startup accelerator that makes (pre-seed!) investments. I don't know if anyone has ever sincerely taken "YC company" to mean a subsidiary of YC. That's like confusing "a Sequoia company" as a subsidiary of Sequoia.
reply