Niko1221/Strata

…less stuff. Overall, IIRC most of them still had a >30% chance of producing wrong/non-working replies. I then tested Strata ( https://github.com/niko1221/strata ) and the results are not even close. You can basically ignore everything in that table and use Strata instead, you can use bigger/less-quantized models and have way better results. Basically any otherwise-decent model degrades noticeably at Q < 5, and KV cache at Q4 also contributes a lot. Still, a 20$/mo Claude subscription is way cheaper/more efficient if you code 2-3h in the evening than running stuff locally, unless you really have hardware than can hold models that get most things right at the first try. You can iterate way f…

If you have enough system RAM (64GB), you can run Qwen 3.8 Flash Next with a GPU as small as 12GB, this repo is pretty amazing: https://github.com/Niko1221/Strata/ My setup is a Ryzen 3600, 90GB DDR4 allocated to a VM (don’t even really need that much, just had it available, special thanks to the year 2024), and an RX 9060XT 16GB ($500 GPU). I get around 40-60 tok/s depending on the quantization level I choose. As you can see, the CPU is nothing special, don’t even need DDR5.

I ran the Strata project with this on 32gb ram 16gb AMD 7900XTX. It goes about 20tok/s off the ssd. I'm guessing 64gb would go to 40tok/s. https://github.com/Niko1221/Strata Quality wise its not fast enough for me to eval. I'll probably spend some tokens on it through openrouter. DeepSeek 4 flash though is a mess compared to GPT Sol 6.1.

Oh you should be running flash next with strata. Just go to stratas github page [0] and it will explain everything. It runs quite fast on 16+64, you should be able to get 50 tok/s or so. It's a 180b model and it feels a lot smarter than most of the other open weight options. As far as knowing "what will fit" in general that's hard to know these days. The moe models are hard to predict and run wildly different in different engines (llama.cpp, vllm, strata likes, etc) but they tend to be the only good options around 16gb vram. Dense models like qwen3.8 27b are only good if they're entirely in vram. Moe models you need the active count + a few experts (although it will run faster the more exper…

Then check out those: https://github.com/prabanta-dev/janas <- this, focuses on Qwen3-Next-80B-A3B. https://github.com/Niko1221/Strata <- Qwen3.8-Flash-Next. I use this (tuned) and I cannot understate how satisfied I am. https://github.com/antirez/ds4 <- DeepSeek 4 family + more.

Not sure why this is downvoted. It is fully possible to run the 180B Owen model on strix halo using strata and even get reasonable speed out of it. https://github.com/Niko1221/Strata

Yep, I agree. You can already run them locally on a top-shelf gamer PC with decent token throughput: https://github.com/Niko1221/Strata That's today. As for tomorrow, consider that literally every chip maker is working on better and better NPUs to boost local performance (Ryzen AI, Intel Ultra, Apple M7, Broadcom PON, Huawei Ascend, Snapdragon X Elite, etc.), and imagine optimized models running on these, yeah, insanely huge potential. As of 2026Q4 I wouldn't bet on any cloud AI, that's for sure.

Take a look at the docs for https://github.com/niko1221/strata for some ideas, particularly https://github.com/Niko1221/Strata/blob/main/docs/MODELS.md#pick-by-memory .

You can get something relatively capable and usable with your configuration and Strata: https://github.com/Niko1221/Strata