Page 1 of 2 12 LastLast
Results 1 to 10 of 18

Thread: oom killer back active after update...

  1. #1

    Exclamation oom killer back active after update...

    Code:
    jdivm04:~ # uname -a
    Linux jdivm04 5.3.18-lp152.66-default #1 SMP Tue Mar 2 13:18:19 UTC 2021 (73933a3) x86_64 x86_64 x86_64 GNU/Linux
    
    jdivm04:~ # lsb_release -a
    LSB Version:    n/a
    Distributor ID: openSUSE
    Description:    openSUSE Leap 15.2
    Release:        15.2
    Codename:       n/a
    Hi,

    Using leap 15.2 and from time in time (usually each 3 months) I ran an update on my linux (zypper ref + zypper up).
    I ran on this linux two databases (IBM Informix + mysql) and in the past I already have lot of issues with OOM KILLER.
    So, I "disabled" it and haven't any problem a couple years, until now.
    The main solution, which I have set and still set is this configuration on sysctl :

    Code:
    jdivm04:/etc/sysctl.d # sysctl -a | grep overcomm
    vm.nr_overcommit_hugepages = 0
    vm.overcommit_kbytes = 0
    vm.overcommit_memory = 2
    vm.overcommit_ratio = 95
    However, three weeks ago I ran my update and then OOM KILLER back in activity, killing with high frequency my databases and others services.
    I have all set fine about my memory configuration and I really don't understand why it still killing them.
    The last time it kill the mysql , at dmesg the message finish with this text:
    Code:
    [  +0.000001] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/,task=mysqld,pid=56172,uid=60
    [  +0.000065] Out of memory: Killed process 56172 (mysqld) total-vm:4673904kB, anon-rss:1218612kB, file-rss:0kB, shmem-rss:8kB
    [  +0.036022] oom_reaper: reaped process 56172 (mysqld), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
    I already tried alternative solutions to deactivate it , like below, with no effect:
    Code:
    echo 0 > /sys/fs/cgroup/memory/memory.use_hierarchy
    mkdir /sys/fs/cgroup/memory/0
    echo 1 > /sys/fs/cgroup/memory/0/memory.oom_control
    As far I remember, before the update I was running kernel 5.3.18-lp152.60 or 63, not sure , now is 66. (since I ran new updates to see if solve this behave)

    Any tips how to deactivate for good the OOM KILLER?

  2. #2
    Join Date
    Feb 2010
    Location
    Germany
    Posts
    4,043

    Wink Re: oom killer back active after update...

    @ceinma:

    The Oracle article “How to Configure the Linux Out-of-Memory Killer” mentions the following –
    The OOM killer can be completely disabled with the following command. This is not recommended for production environments, because if an out-of-memory condition does present itself, there could be unexpected behavior depending on the available system resources and configuration. This unexpected behavior could be anything from a kernel panic to a hang depending on the resources available to the kernel at the time of the OOM condition.
    Code:
    sysctl vm.overcommit_memory=2
    echo "vm.overcommit_memory=2" >> /etc/sysctl.conf
    <https://www.oracle.com/technical-res...om-killer.html>

  3. #3
    Join Date
    Jun 2008
    Location
    West Yorkshire, UK
    Posts
    3,572

    Default Re: oom killer back active after update...

    Yes, I also find that settings are occasionally overwritten during an update but what is the problem with increasing available swap memory so that oom killer is never invoked?

  4. #4

    Arrow Re: oom killer back active after update...

    Quote Originally Posted by dcurtisfra View Post
    @ceinma:

    The Oracle article “How to Configure the Linux Out-of-Memory Killer” mentions the following –

    Code:
    sysctl vm.overcommit_memory=2
    echo "vm.overcommit_memory=2" >> /etc/sysctl.conf
    <https://www.oracle.com/technical-res...om-killer.html>

    Hi @dcurtisfra ,
    Yes, if you read my message will notice I already have this configuration set and not working anymore.

  5. #5

    Default Re: oom killer back active after update...

    Quote Originally Posted by john_hudson View Post
    Yes, I also find that settings are occasionally overwritten during an update but what is the problem with increasing available swap memory so that oom killer is never invoked?

    I also tried that doubling my swap to get the memory calculation of CommitLimit vs Committed_AS with considerable free space and even that doesn't work.
    I begin to monitor the /proc/meminfo/Committ* to have sure about my memory usage and is far away to get close of consume all them , check the graph below for the last 7 days. The lowers at Commit_AS is when the oomkiller killed some of my databases...


    https://i.imgur.com/tOmsGQg.jpg


    (tried to embed the image, but this editor give me an error)




    It just killed again my IBM Informix database... errrr I hate this OOM

  6. #6
    Join Date
    Feb 2010
    Location
    Germany
    Posts
    4,043

    Default Re: oom killer back active after update...

    Quote Originally Posted by ceinma View Post
    I also tried that doubling my swap to get the memory calculation of CommitLimit vs Committed_AS with considerable free space
    It seems that, the newest Kernel doesn't allow the OOM-Killer to be disabled …

  7. #7
    Join Date
    Jun 2008
    Location
    San Diego, Ca, USA
    Posts
    13,285
    Blog Entries
    2

    Default Re: oom killer back active after update...

    In this article I wrote long ago, I described how to shift resources from the system to networking.
    You can use the methods I describe to read various parameters and possibly do the reverse to shift more resources to your system (I don't recommend permanently, maybe do so only for the specific task).

    https://sites.google.com/site/4techs...nnection/linux

    Also,
    In the article I wrote how to use the Free tool to analyze memory,
    I also included the command to "Clear memory buffers and cache" which I'm guessing should be very useful in your case. The command should be harmless although I would of course caution not to do this in the middle of some long calculation. Wait until your system is quiet, and the main effect is that any data that might have been in memory will simply have to be retrieved again or re-calculated.

    https://en.opensuse.org/User:Tsu2/free_tool

    Also, beware configuring too much swap, swap requires using space in RAM to point to the nee locations on disk so you can actually be making your memory starvation even worse by creating more swap.

    Needless to say,
    If a system update is causing memory problems, it's likely a warning that your system has been provisioned to the max, and could use a RAM upgrade.

    TSU
    Beginner Wiki Quickstart - https://en.opensuse.org/User:Tsu2/Quickstart_Wiki
    Solved a problem recently? Create a wiki page for future personal reference!
    Learn something new?
    Attended a computing event?
    Post and Share!

  8. #8
    Join Date
    Feb 2010
    Location
    Germany
    Posts
    4,043

    Cool Re: oom killer back active after update...

    @ceinma:

    Quote Originally Posted by tsu2 View Post
    If a system update is causing memory problems, it's likely a warning that your system has been provisioned to the max, and could use a RAM upgrade.
    Yes, exactly –
    • The workarounds I pointed to are really only a “piece of plaster on the wound” – to really address the issue the affected machines urgently need more memory (if physically possible) …
    • If the machines are all provisioned with the maximum amount of memory physically possible then, you'll have to look at purchasing more servers to spread the application load …


    I didn't really want to write this because, it's the “normal” method to deal with such issues …

  9. #9

    Default Re: oom killer back active after update...

    So , for me the unique and easy workaround is increase , again, my swap.
    Before I had 4GB of swap and changed to 8GB, without success to solve this .
    I will try again.

    But I really want to found a kernel documentation saying the OOM killer is now 100% of the time active.
    I'm looking and found nothing.

    I will add more 4GB to my swap and see what happen, however, I think will change nothing...

    I'm avoiding to grow the RAM since this is a VM sharing with others VMs , I don't want to get resources unnecessary.

  10. #10

    Default Re: oom killer back active after update...

    I don't mentioned before, but I also have this settings for a long time :

    Code:
    vm.vfs_cache_pressure = 10
    vm.dirty_background_ratio = 1
    vm.dirty_expire_centisecs = 60000
    vm.dirty_ratio = 1
    
    
    vm.swappiness = 0
    Now I changed the swappiness from 0 to 5 , let's see if change any behave of this **** OOM .

Page 1 of 2 12 LastLast

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •