{"id":18,"date":"2007-09-02T02:32:50","date_gmt":"2007-09-01T23:32:50","guid":{"rendered":"http:\/\/www.held.org.il\/blog\/?p=18"},"modified":"2007-09-02T02:32:50","modified_gmt":"2007-09-01T23:32:50","slug":"out-of-memory-in-linux-or-oom-killer","status":"publish","type":"post","link":"http:\/\/www.held.org.il\/blog\/2007\/09\/out-of-memory-in-linux-or-oom-killer\/","title":{"rendered":"Out of memory in Linux (or: OOM Killer)"},"content":{"rendered":"

Do you remember DOS games or apps quitting with the reason 'not enough memory'? It rarely happens today, because physical ram is not the limit - physical ram + swap, or virtual memory size<\/strong> is the limit.<\/p>\n

So what happens when we're out of virtual memory? Linux doesn't crash. Instead, it looks for the best process(es) to kill. \"Best\" means: killing the process will free up lots of memory, but will do the least damage possible (i.e. don't kill init, hardware-bound process, or a long-running software).<\/p>\n

<\/p>\n

Who do you want to assassinate today? (tm)<\/h3>\n

The relevant part in the kernel is called OOM Killer<\/strong> , written by Rik Van-Riel<\/a>, a Linux memory management guru.<\/p>\n

OOM Killer gives score to each process, which can be seen in \/proc\/<pid>\/oom_score<\/em>. Higher score means better candidate for killing. The scoring algorithm as of 2.6.22, follows (Please comment if something here is not correct):<\/p>\n

    \n
  1. Take the number of bytes of virtual memory<\/strong> the process uses, plus half of the bytes of the children<\/strong>.<\/li>\n
  2. Divide<\/strong> by sqrt(cpu time seconds<\/strong>)<\/li>\n
  3. Divide<\/strong> by sqrt(run time seconds<\/strong> - time since the program started)<\/li>\n
  4. Multiply<\/strong> by two, if process is nice (nice>0)<\/li>\n
  5. Divide<\/strong> by four if it's owned by root<\/strong><\/li>\n
  6. Divide<\/strong> by four if it's accessing hardware<\/strong><\/li>\n
  7. Divide<\/strong> by eight if the process' memory does not overlap <- I'm not sure about that, I think it's SMP-related.<\/li>\n
  8. Shift<\/strong> by oomkilladj<\/strong> (explanation follows)<\/li>\n<\/ol>\n

    Adjusting the killer's scoring systems<\/h3>\n

    And to the cool part: oomkilladj<\/strong> is a way to adjust the process' score! \/proc\/<pid>\/oom_adj<\/em> is zero by default, but can be -17 to 15. You can play with it and see oom_score<\/em> changes immediately!<\/p>\n

    The value '2' would shift-left the score twice (multiply by four), while the value '-1' would shift-right the score once (divide by two).<\/p>\n

    -17 is a special value meaning never kill <\/strong>this process.<\/p>\n

    Seeing it live..<\/h3>\n

    And just to make this post complete, here's a little script which prints the list of processes and their oom_score, ordered by oom_score ascending:<\/p>\n

    #!\/bin\/bash\nfor i in `find \/proc -maxdepth 1 -type d`; do\nif [ -f $i\/oom_score ]; then\necho -n \"$i \";\ncat $i\/oom_score;\nfi;\ndone | sort -n -k2<\/pre>\n
    Refs<\/h3>\n