ggml-org/llama.cpp

…the model into making reasoning self corrections that only got worse with longer sessions (more details about this issue: https://github.com/ggml-org/llama.cpp/issues/24181#issuecomment-4792181456 ) No inference engine is perfect, but I feel that llama.cpp is the most reliable way to run language models locally.

…ntext on a 5070ti with 16 GB of VRAM (using the Unsloth 2-bit quant[0]). This is via a .bat file on Windows 11. I'm getting about 50-60 tokens/second and the quality is much higher than Qwen 3.6 27B. I'm using llama.cpp[1]: llama-server.exe ^ -m "Qwen3.8-27B-UD-Q2_K_XL.gguf" ^ --presence-penalty 0.0 ^ --repeat-penalty 1.0 ^ --fit-ctx 128000 ^ -ctk q4 0 ^ -ctv q4 0 ^ --reasoning-budget -1 ^ --chat-template-kwargs "{\"preserve thinking\": true}" ^ --host 0.0.0.0 ^ --port 8033 [0] https://huggingface.co/unsloth/Qwen3.8-27B-GGUF (UD-Q2_K_XL) [1] https://github.com/ggml-org/llama.cpp/releases Instructions if you want to do the same: 1. download two files llama-b10434-bin-win-cuda-13.3-x64.zi…

llama.cpp works pretty well for me on the Framework 13 laptop, but the current era of "move fast, break things, rarely fix" (sorry, that's how it feels), bites here quite a bit. Two examples: - https://github.com/ggml-org/llama.cpp/pull/25863 Someone's few lines change broke the native (ROCm) support for the AMD GPU inside Framework (and other integrated systems), and any rollback or proper fix is pending for almost a month. Fortunately there's workaround (switching to Vulkan rather than ROCm devices), but both the way the bug was introduced and the way it is not fixed just doesn't give much confidencen - LM Studio is using llama.cpp internally for GGUF, they ship their own build with the…

I use llama.cpp w/ llama-swap https://github.com/ggml-org/llama.cpp https://github.com/mostlygeek/llama-swap

I just build llama.cpp from scratch on the PR that has MTP drafters. https://github.com/ggml-org/llama.cpp/pull/23398 Please don't use Ollama, it's a bad actor in the OSS community.

https://github.com/ggml-org/llama.cpp/blob/master/grammars/README.md llama.cpp supports grammar limiting using either GBNF or json schema (It just translate it to GBNF behind the scenes I think). So I have my harness generate a tool schema on the fly (based on what tools are possible for the current task) and pass it in at request time.

…better but fall off a bit in the real world.) This will get you close to like ~Sonnet 4 though: Grab a recent win-vulkan-x64 build of llama.cpp here: https://github.com/ggml-org/llama.cpp/releases - llama.cpp is the engine used by Ollama and common wisdom is to just use it directly. You can try CUDA as well for a speedup but in my experience Vulkan is most likely to "just work" and is not too far behind in speed. For best quality, download the biggest version of Qwen 3.5 27B you can fit on your 4090 while still leaving room for context and overhead: https://huggingface.co/unsloth/Qwen3.5-27B-GGUF - I would try the UD-Q5_K_XL but you might have to drop down to Q5_K_S. For best speed, you co…

Have you tried compiling llama.cpp with Unified Memory Access[1] so your iGPU can seamlessly grab some of the RAM? The cmake boolean is prefixed with CUDA but this is not CUDA specific. It made a pretty significant difference (> 40% tg/s) on my Ryzen 7840U laptop. 1 - https://github.com/ggml-org/llama.cpp/blob/master/docs/build.md#unified-memory

It's a lot better than it used to be and honestly way more powerful. If you haven't looked lately, you should https://github.com/ggml-org/llama.cpp/tree/master/tools/server

I'm using the default llama-server that is part of Gerganov's LLM inference system running on a headless machine with an nVidia 16GB GPU, but Ollama's a bit easier to ease into since they have a preset model library. https://github.com/ggml-org/llama.cpp

I switched from llama.cpp to vLLM because of prompt cache bugs in qwen/gemma models This is a good starting issue with a bunch of linked/related https://github.com/ggml-org/llama.cpp/issues/22746

Hey - I use the same, w/ both gemma4 and gpt-oss-*; some things I have to do for a good experience: 1) Pin to an earlier version of codex (sorry) - 0.55 is the best experience IME, but YMMV (see https://github.com/openai/codex/issues/11940 , https://github.com/openai/codex/issues/8272 ). 2) Use the older completions endpoint (llama.cpp's responses support is incomplete - https://github.com/ggml-org/llama.cpp/issues/19138 )