noonghunna/club-3090

With spec decode should easily get to >100tps on my dual 3090s qwen 3.5 27b was running at around 110tps using the config from https://github.com/noonghunna/club-3090 make that 200tps on a single 5090, 4x faster than opus https://x.com/radixark/status/2088285681131110446 devs about to get handed a two 5090 box each and told to max that out

This is not my experience at all. Even the Nous Research guys have stated that "Qwen3.6-27B is the canonical local model to use Hermes Agent with" [ https://old.reddit.com/r/LocalLLaMA/comments/1sz2y76/ama_with_nous_research_ask_us_anything/oiynr8i/ ]. I am finding the same when used with Pi and OpenCode. Gemma will just stop mid-tool call. It's been slower and I've had to reduce context size to run it. Qwen3.6 27b has been rock solid using club 3090's single card setup for agentic use -- https://github.com/noonghunna/club-3090/blob/master/docs/SINGLE_CARD.md

small native inference engine optimized first for DeepSeek V4 Flash (including the experimental vision model), DeepSeek V4.1 Flash (Metal, and text inference on CUDA), and additionally GLM 5.2 and 5.3, GLM 5.3 Flash and DeepSeek V4 PRO, and Qwen3.8 Flash Next (Metal and CUDA) This is local targeting high end consumer hardware like DGX Spark or AMD Ryzen AI Halo. For our mere mortals that were kids not long ago and can't really believe we've got our hands on a x090 series targeting Qwen3.8 27b, https://github.com/noonghunna/club-3090 is the way to go. I'm maintaining a web frontend for this, trying to at least. You can follow it here: https://github.com/gchamon/club-3090-server

I beg to differ. Have a look at this repo with single/double 3090 optimized configs for Qwen and Gema models: https://github.com/noonghunna/club-3090

This is a good place to start reading about dual gpus. https://github.com/noonghunna/club-3090/blob/master/docs/DUAL_CARD.md

If I started today, with building a server, I'd jump right into verified set-ups and writeups, like this one: https://github.com/noonghunna/club-3090 You can find info about running a patched version of vllm for 1x24gb, 2x and 4x. There's also quite a few "blackwell" subreddits, where people seem to share a lot of substantial information, if you're going the 6000 route.

I can see that and I don't know your setup, but there are people pushing >70t/s with MTP on a single 3090, with big contexts still >50t/s. 64k is not a lot for agentic coding, and IIRC 128k with turboquant and the likes should be possible for you. r/LocalLLM/ and r/LocalLLaMA/ are worth a visit IMO. EDIT: just found this recipe repo, may wanna give it a go: https://github.com/noonghunna/club-3090 EDIT-2: this can also shave off a lot of context need for tool calling -> https://github.com/rtk-ai/rtk

I run it with Llama.cpp on my RTX 3090. Also using the same Unsloth model. My config is similar to: https://github.com/noonghunna/club-3090/blob/master/docs/engines/LLAMA_CPP.md I need to try out some of the other set ups mentioned in this repo for increased TPS.