IMHO, the author could have done two things better:
- vllm instead of llama.cpp. With NVIDIA HW, there is huge difference in multi-user loads and caching with vllm; when he was complaining about what happens when more than one user uses the model, and about losing caching, I was "well, duh".
- The budget he used for a single card could have instead be put to far, far better use with SPARKs. I have access to a cluster of 2 x GX10 - total cost less than half what he paid, even today - and I am running vllm and Deepseek v4 Flash. The difference compared to any Qwen is tremendous - I've NEVER seen it loop, and in all my experiments so far, it's the most Sonnet-y model I've ever tried (antirez seems to agree, hence his ds4 fork).
Performance: 2K t/s prefill ( very useful for feeding tons of source code into its massive context window ) and around 50-60 tg/s in my coding sessions in the pi.dev harness. With the money the author paid, he could have bought 4 GX10s, and double both numbers ( vllm basically scales almost linearly with tensor parallelism ).
We did run vLLM on the 3090s — measured ~3 tok/s slower on generation for our single-to-few-user pattern, plus less flexibility on quant and slower startup (actual minutes vs single digit seconds). We may do more with it again in the future - there isn't unlimited time for us to tinker, I'm sharing our journey (so far) and reasoning.
It's the right call for concurrent batched serving (barrkel's point downthread is spot on), but for how we use it llama.cpp is still better for us.
The Spark/GX10 route is a genuinely different bet though and appreciate you sharing your numbers. At the time (several months ago) the consensus was that GX10s were for fine-tuning only, and the numbers were severely low.
..and the card was never about replacing a Claude Max sub. For the workloads we actually bought it for, it's giving us 140-200 tok/s (which matters).
I hear you on the insane amount of time vllm takes to launch (atlas is a move in the right direction in that regard).
But mostly I wanted to raise awareness to readers of your article that no, if you want to do inference, paying 15K for a single 96GB card almost certainly makes no sense. Buy 4 GX10s with the same money, and enjoy dramatically better models and user scalability.
Regardless - thanks for putting the effort to share your findings! I keep postponing doing the same... there's tons of things everyone is re-discovering on their own.
wanna chime in, recently tried vLLM to consume a NVFP4 Gemma4 safetensor model and see how the batching can show up in nice t/s numbers.
it's slow to start, it's Linux only, it doesn't like WSL much, ended up with either old or nightly container builds, I more or less have given up.
Appreciate how llama.cpp simply works and does things fast and obvious
As soon as I met Forth, I felt I had to hack my own (https://github.com/ttsiodras/MiniForth/) - and did so over a single week, two years ago. I targeted the Blue Pill and the original Arduino, but developed via cross-compilers so that I could test my code in the host.
I became so obsessed with the project that I was looking forward to tinkering with it after coming back from work every day; so it was hacked in 5 evenings and a weekend.
It was that much fun, to build a Forth.
> I became so obsessed with the project that I was looking forward to tinkering with it after coming back from work every day; so it was hacked in 5 evenings and a weekend. It was that much fun, to build a Forth.
Yep. In my computer architecture class as a freshman, we were supposed to do a final project of our choosing for the LC-3 (a RISC instruction set with emulator). I had dabbled briefly in Forth (with the RedPower 2 Minecraft mod) and thought it'd be fun to implement one. It ate up essentially all of my free time that semester: There was one morning where I only realized I'd stayed up all night when the sun started to come up and I finally checked the clock.
I've never found a practical use for Forth in my "real" life, but building one from scratch was an experience almost best described in religious terms. It was a pure distillation of Fred Brooks's description of programming:
> The programmer, like the poet, works only slightly removed from pure thought-stuff. He builds his castles in the air, from air, creating by exertion of the imagination. Few media of creation are so flexible, so easy to polish and rework, so readily capable of realizing grand conceptual structures.
> Yet the program construct, unlike the poet's words, is real in the sense that in moves and works, producing visible outputs seperate from the construct itself. It prints results, draws pictures, produces sounds, moves arms. The magic of myth and legend has come true in our time. One types the correct incantation on the keyboard, and a display screen comes to life, showing things that never were nor could be.
Fully agree. About 40,000 eons ago I wrote a Forth compiler in ZX81 basic. It went from taking 19 seconds to print 1000 numbers on the screen to instantly. I also learned a great lesson: I looked at the machine code it generated (it had a pinhole optimizer because the output was PUSH HL, PUSH DE, POP DE, POP HL everywhere), and I thought it was perfection - impossible to improve. The day later I came back and made it much faster.
This is why I come to HN - brilliant ideas that I should have thought of before, but didn't!
I just changed my server, and uninstalled the - now truly useless - fail2ban. I use SSH keys of course, but without fail2ban my server's logs were constantly flooded with hacking attempts.
No longer - wireguard for the win. Thank you, chlorion!
"Affected products: ... Up to (excluding) 152.0.7977.82"
.82 is fixed.