Sure. The fastest small coding model is probably Mellum2 12B-A2.5 by Jetbrains. It matches or beats all Qwen models in this class.
Can even run on a notebook CPU and comes in Base (best for FIM), Instruct and Thinking variants. mradermacher has imatrix quants for people who can't run it at Q8.
IQ4 should fit, but even if it doesn't, llama.cpp has options to partially offload models to system memory.
I'm able to run qwen 3.6 35b a3b with my rtx 2060 super which also has 8gb of vram. It's a very capable model, not as smart as a frontier model but it is still very useful.
I use an 6bit quant and get around 20 tokens per second.
A small model, yes! But not necessarily a good model.
With the additional caveat that I don't know whether that specific card is supported by modern drivers.
You'd be looking at one in the 6B or 7B parameters range at FP8. Or smaller. It's been quite some time since a recognizable company in the AI space released a model that small. You can try larger model that has been quantized down to that size, but they don't always fare well with that.
Modern text-to-speech and speech-to-text models also fit well into modest amounts of VRAM.
You're arguing for a very specific range of weights but many slightly smaller and slightly larger models have been released including QAT and MoE versions.
An old nVidia brand card with 8GB is more than enough to see those models running at usable speeds and accuracy.
The software that won the competition seems to map the configuration space of an origami crease pattern using techniques from statistical mechanics - pretty snazzy!
I love this -- I'll have to do something like that for my site. I always liked the big initials on the start of a paragraph. Though it feels a bit more prose-applicable than for non-fiction writing.
I heard about lottocracy/sortition for the first time not long ago and I quite like the idea. The last time was when I heard a professor talk about it, and I was recommended reading the book "Lottocracy: Democracy Without Elections" by Guerrero [0].
I just had a nice trip to Venice and I was curious about it's history. Supposedly, the Venice republic lasted almost 1000 years, basically from after the fall of Rome to Napoleon based on a weird lottery system for choosing the Doge.
I've never read up on the republic of Venice, but after quickly scanning the Wikipedia article on its election procedure... that is a strangely large number of voting rounds and lotteries.
Before you get too excited about this just imagine the average line of people at the DMV or the Grocery store and now imagine that those people are in charge of the lives of hundreds of millions. If you think HOAs are bad, you aint seen nothing yet.
The current system of oligarch patronage is bad, but at least it keeps the train mostly on the rails.
But aren't most HOA horror stories based on people who'd been running them for years if not decades, and only end happily when someone replaces those entrenched in power with new people?
But such groups are almost invariably coordinated. In a legislature based on sortition, there will be a percentage of busybodies/ assholes/ opportunists but they'll have a coordination problem, opponents, and term limits acting to restrain them.
Term limits incentivize a deep state exactly one layer removed from those to which the limits apply, as a repository of institutional knowledge about how things actually get done.
This seems rational. We on't have term limits int he US Congress and it doesn't seem any the better for it.
Japan, a heavily bureaucratized country, systematically moves junior and mid-tier staff around in some departments to minimize the possibility of nest-feathering and empire-building, although I would not say it's perfect by a long way.
We do have term limits for positions like the presidency, and what we see is a perpetual power structure one layer removed, in the party system, which effectively chooses who we're permitted to vote for.
Introducing term limits only forces the wealth and power to change it's face periodically. It is addressing a symptom, not the cause.
It's a potentially big problem for sure. It reminds me of stories I've heard about the public education system in some of the Scandinavian countries. From what I remember off the top of my head, Finland has a system where private educational facilities do not exist. Meaning that, if rich or otherwise elite people want their kids to receive a good education, they need to support the public education facilities their own kid attends. I quite like this idea that everyone is nudged towards helping everyone else, even if they mostly care about their own family and friends.
Similarly in a lottocracy you'd want everyone to be a capable leader when their name is picked from the hat. As the professor I listened to put it, lottocracy makes you think what a democracy really values. Is it about everyone's voice being heard, or is there another goal we should care about more? Not an easy question to answer.
Yes, I suppose there exists an egalitarian and well adjusted hypothetical society where we could find good leaders by random draw. I just don't think we live in anything resembling that society and I'm not sure whether such a society is possible once you reach a certain population size.
I think it's a nice idea, but I'm not sure how we get from here to there
> Yes, I suppose there exists an egalitarian and well adjusted hypothetical society where we could find good leaders by random draw.
If you can find good leaders by random draw, that means the average citizen is a good leader, which would seem to suggest that the average citizen should be a reasonable an hard-to-dupe judge of good leaders, and therefore that elections also work well.
If elections don't work well to select leaders, that's a pretty good piece of evidence that sortition won't, either.
OTOH, the particular failures of sortition and elections may be different, and using a system where both are used for different veto points might be net less problematic than either alone. Consider a bicameral legislature with one house chosen by elections and the other by sortition, for instance.
(OTOH, there is plenty of solid evidence in comparative government of how to do electoral democracy better and people in the US don't seem too interested in that, which is probably a better focus for immediate reform than relatively untested, on a large scale, ideas about avoiding electoral democracy.)
Bit of a nerd-snipe, but I wonder about the idea of sortition of a set of candidates -- say 200 -- out of a larger voting pool, and then voting for one of the randomly selected candidates.
Then you get "at least approx. top 1%" -- but it's still not necessarily an entrenched elite.
Agreed, I'm not sure if it can be made to work either. I have an inkling of a thought that instead of an egalitarian society being required for lottocracy to work, an egalitarian society can be created using lottocracy. But it's just a thought. Hopefully that book holds something close to an answer, but I'll see :)
