rfjakob/earlyoom

Possibly not a solution for your use case, but I use earlyoom[0]. When memory crosses a threshold value, it will kill the worst offender. Has some knobs to tweak: what threshold level you want, which programs to keep safe, which to prioritize, etc. Never had to touch the configuration, out of the box has been sufficient for me. I use it when I am doing exploratory work and might materialize something too big. It has always killed the Python process or terminal program which did something silly. [0] https://github.com/rfjakob/earlyoom

OS are designed to fully exploit available resources, Linux tries its best before triggering an OOM kill. I recommend using the earlyoom if you want more aggresive oom kill: https://github.com/rfjakob/earlyoom The README contains a lot of interesting information.

For OOM, there's earlyoom [1] and systemd-oomd [2]. They usually work out of the box [1]: https://github.com/rfjakob/earlyoom [2]: https://man.archlinux.org/man/systemd-oomd.8

Install oomd or earlyoom. https://github.com/facebookincubator/oomd/ https://github.com/rfjakob/earlyoom

If using systemd-oomd, you can launch Firefox into it's own cgroup / systemd.scope, that has memory pressure control settings set to not kill it. ManagedOOMPreference=avoid. https://www.freedesktop.org/software/systemd/man/latest/systemd.resource-control.html#Memory%20Pressure%20Control There's a variety of oom daemons. bustd is very lightweight & new. earlyoom has been around a long time, and has an --avoid flag. https://github.com/rfjakob/earlyoom?tab=readme-ov-file#preferred-processes Your concerns are very addressable.

I can recommend earlyoom ( https://github.com/rfjakob/earlyoom ). Instead of freezing or crashing your system this tool kills the memory eating process just in time (in this case duckdb). This allows you repeat with smaller chunks of the dataset, until it fits into your mem.