I am not-incredibly-surprised that none of these many talks about swap have mentioned kubernetes. Kubernetes does not solve this problem conclusively, but it does solve it to a huge extent. Your Application Cgroups have lower RAM priority than system tasks. You allocate enough RAM for system-level tasks in kubernetes ( --system-reserved and --kube-reserved). You run your application tasks in a fixed reservation, and any that are over their reservation are first to be killed. If application tasks are repeatedly getting killed, your node is faulty and gets restarted. All of this is done with little fuss. At some higher level, faulty processes are easy to identify because they are the most evicted (And if your system-level daemons are what's at fault, it gets really obvious because of how unobvious the application is)
https://kubernetes.io/docs/tasks/administer-cluster/reserve-...