kyuz0/amd-strix-halo-toolboxes

yes. I build a docker file from this: https://github.com/kyuz0/amd-strix-halo-toolboxes It has all the backends you want. Just a script rebuilds it. Then I just put it in a docker compose and whenever we want to update it, we pull the repo and rebuild it. I just a models.ini like: version = 1 ``` [*] n-gpu-layers = 999 threads = 28 flash-attn = on mlock = off mmap = off fit = off warmup = on jinja = true direct-io = on cache-prompt = true cache-reuse = 256 models-max = 1 models-autoload = 1 timeout = 600 ctx-checkpoints = 256 cache-ram = 0 mlock = on kv-unified = on no-context-shift = on swa-full = on [Q36-35B-A3B] model = Qwen3.6-35B-A3B-UD-Q8_K_XL.gguf mmproj = mmproj-BF16.gguf rope-sca…

I realize it does not address the OP security concerns, but I'm having success running rocm containers[0] on alpine linux specifically for llama.cpp. I also got vLLM to run in a rocm container, but I didn't have time to to diagnose perf problems, and llama.cpp is working well for my needs. [0] https://github.com/kyuz0/amd-strix-halo-toolboxes

If you're on Strix Halo, check out Donato's prebuilt toolboxes for ROCm with RADV or Vulkan: https://github.com/kyuz0/amd-strix-halo-toolboxes It takes all the work out of it, you just start llama-server in the container context and you're off doing inference without having to figure out dependencies.

No. In Linux, you can ignore the BIOS (well, set it to 512MB) and set GTT to allow almost the entirety of memory for GPU. If you don't run a desktop GUI (and why would you), you don't need memory for much of anything else. Just a few GB for the OS and llama.cpp. These are roughly the settings I use: https://github.com/kyuz0/amd-strix-halo-toolboxes#kernel-parameters-tested-on-fedora-42

If you just want to run models, most of TFA is taking the scenic route. All you really need is podman, toolbx, and the Strix Halo toolbox images from https://github.com/kyuz0/amd-strix-halo-toolboxes . Then you just download your ggufs and hand them to llama-server. Yes, there are other solutions that are a bit more hand-holdy, but if you already know how to use docker/podman and just want to get something working in an evening, this works too.

I took my setup from here: https://github.com/kyuz0/amd-strix-halo-toolboxes Still lot to learn, but after a while you have something like Qwen3-Coder-Next-Q8_0 running and - at least for me - it works quite well, both as ChatGPT like chat-interface using llama.cpp and as coding agent

… messages. But then i often start a new chat. Lemonade-server runs with llama.cpp, vllm seems to be scaling better thoug, but is not so easy to set up. Unsloth GGUFs are great resource for models. Also for Strix Halo check out kyuz0 repositorIES! Also has image gen. I didnt try those yet. But the benchmarks are awesome! Lots to learn from. Framework forum can be useful, too. https://github.com/kyuz0/amd-strix-halo-toolboxes Also nice: https://llm-tracker.info/ It links to some benchmark site with models by size. I prefer such resources, since it is quite easy to see which one fit in my RAM (even though i have this silly thumbrule Billion Token ≈ GB RAM). Btw. even a AMD HX 370 with non …