Tuning the oom killer / low memory killer

direc85 · 15 August 2022 17:34

Disclaimer: Changing the values listed in this post may and will affect the stability of your device, and can lead into unusable state or data loss. Fiddle with the settings at your own risk.

I have a Sony Xperia 10 III and I’m suffering a lot from different kinds of things, which I assume are caused by the operating system killing processes left and right. I had a hunch that the system is too aggressive on this, so I finally decided to dig deeper.

It turns out there are two actors in play:

in-kernel oom killer
lowmemorykiller (aka. lkmd)

Exercising some trial and error, I figured out a command to view/log the lowmemorykiller actions:

dmesg -w | egrep "lowmemorykiller|oom_reaper|^ {19}.*|Out of memory"

I then let the log roll via SSH to the phone, and started switching between (large-ish) Android applications and to browse heavy sites with both Browser and Fennec (Firefox, but from F-Droid). The results contain a surprisingly high amount of kills:

[142202.722417] lowmemorykiller: Killing 'ec_fdroid:tab15' (29069), adj 900,
                   to free 21kB on behalf of 'kswapd-1:0' (102) because
                   cache 703724kB is below limit 945120kB for oom_score_adj 529
                   Free memory is 944692kB above reserved.
                   Free CMA is 72856kB
                   Total reserve is 92268kB
                   Total free pages is 346384kB
                   Total file cache is 781924kB
                   Slab Reclaimable is 110948kB
                   Slab UnReclaimable is 421020kB
                   Total Slab is 531968kB
                   GFP mask is 0x0
                   Kernel Memory Reclaimable is 72732kB
                   Free Swap 690700kB
                   queue len is 1 of max 1 reason:0x1 margin:45315
[142202.747413] oom_reaper: reaped process 29069 (ec_fdroid:tab15), now anon-rss:0kB, file-rss:0kB, shmem-rss:24kB
[142215.850115] lowmemorykiller: Killing 'AndroidUI' (28891), adj 900,
                   to free 37kB on behalf of 'kswapd-1:0' (102) because
                   cache 681388kB is below limit 945120kB for oom_score_adj 529
                   Free memory is 879460kB above reserved.
                   Free CMA is 72856kB
                   Total reserve is 92268kB
                   Total free pages is 258908kB
                   Total file cache is 763884kB
                   Slab Reclaimable is 110716kB
                   Slab UnReclaimable is 421788kB
                   Total Slab is 532504kB
                   GFP mask is 0x0
                   Kernel Memory Reclaimable is 94960kB
                   Free Swap 690716kB
                   queue len is 1 of max 1 reason:0x1 margin:23446
[142215.871661] oom_reaper: reaped process 28891 (AndroidUI), now anon-rss:0kB, file-rss:0kB, shmem-rss:816kB
[142224.781626] lowmemorykiller: Killing 'nec_fdroid:tab0' (29462), adj 905,
                   to free 90272kB on behalf of 'kswapd-1:0' (102) because
                   cache 674628kB is below limit 945120kB for oom_score_adj 529
                   Free memory is 916792kB above reserved.
                   Free CMA is 72876kB
                   Total reserve is 92268kB
                   Total free pages is 305676kB
                   Total file cache is 757644kB
                   Slab Reclaimable is 109128kB
                   Slab UnReclaimable is 422376kB
                   Total Slab is 531504kB
                   GFP mask is 0x6000c0
                   Kernel Memory Reclaimable is 85544kB
                   Free Swap 690716kB
                   queue len is 1 of max 1 reason:0x2 margin:35133
[142224.794389] oom_reaper: reaped process 29462 (nec_fdroid:tab0), now anon-rss:0kB, file-rss:0kB, shmem-rss:24kB
[142241.906141] lowmemorykiller: Killing 'ssioncontroller' (29519), adj 905,
                   to free 73984kB on behalf of 'kswapd-1:0' (102) because
                   cache 656088kB is below limit 945120kB for oom_score_adj 529
                   Free memory is 858552kB above reserved.
                   Free CMA is 72892kB
                   Total reserve is 92268kB
                   Total free pages is 277104kB
                   Total file cache is 739136kB
                   Slab Reclaimable is 108956kB
                   Slab UnReclaimable is 422016kB
                   Total Slab is 530972kB
                   GFP mask is 0x6000c0
                   Kernel Memory Reclaimable is 55388kB
                   Free Swap 690716kB
                   queue len is 1 of max 1 reason:0x2 margin:28112
[142241.927463] oom_reaper: reaped process 29519 (ssioncontroller), now anon-rss:0kB, file-rss:0kB, shmem-rss:84kB
[142241.969612] lowmemorykiller: Killing 'AndroidUI' (29389), adj 900,
                   to free 39kB on behalf of 'kswapd-1:0' (102) because
                   cache 642744kB is below limit 945120kB for oom_score_adj 529
                   Free memory is 857128kB above reserved.
                   Free CMA is 72892kB
                   Total reserve is 92268kB
                   Total free pages is 284908kB
                   Total file cache is 725792kB
                   Slab Reclaimable is 108972kB
                   Slab UnReclaimable is 422016kB
                   Total Slab is 530988kB
                   GFP mask is 0x0
                   Kernel Memory Reclaimable is 46664kB
                   Free Swap 690716kB
                   queue len is 1 of max 1 reason:0x1 margin:29937
[142241.991278] oom_reaper: reaped process 29389 (AndroidUI), now anon-rss:0kB, file-rss:0kB, shmem-rss:776kB
[142248.117695] lowmemorykiller: Killing 'ec_fdroid:tab23' (29796), adj 905,
                   to free 22kB on behalf of 'kswapd-1:0' (102) because
                   cache 631292kB is below limit 945120kB for oom_score_adj 529
                   Free memory is 765464kB above reserved.
                   Free CMA is 72900kB
                   Total reserve is 92268kB
                   Total free pages is 192084kB
                   Total file cache is 714296kB
                   Slab Reclaimable is 108444kB
                   Slab UnReclaimable is 422068kB
                   Total Slab is 530512kB
                   GFP mask is 0x0
                   Kernel Memory Reclaimable is 47832kB
                   Free Swap 690716kB
                   queue len is 1 of max 1 reason:0x1 margin:6729
[142248.122556] lowmemorykiller: Killing 'AndroidUI' (29728), adj 900,
                   to free 40kB on behalf of 'kswapd-1:0' (102) because
                   cache 629924kB is below limit 945120kB for oom_score_adj 529
                   Free memory is 768516kB above reserved.
                   Free CMA is 72900kB
                   Total reserve is 92268kB
                   Total free pages is 195552kB
                   Total file cache is 712928kB
                   Slab Reclaimable is 108444kB
                   Slab UnReclaimable is 422068kB
                   Total Slab is 530512kB
                   GFP mask is 0x0
                   Kernel Memory Reclaimable is 46912kB
                   Free Swap 690716kB
                   queue len is 1 of max 1 reason:0x1 margin:7722
[142248.129693] oom_reaper: reaped process 29796 (ec_fdroid:tab23), now anon-rss:0kB, file-rss:0kB, shmem-rss:24kB
[142248.139424] lowmemorykiller: Killing 'com.Slack' (28697), adj 199,
                   to free 162964kB on behalf of 'HwBinder:3801_1' (3969) because
                   cache 623316kB is below limit 852960kB for oom_score_adj 147
                   Free memory is 796204kB above reserved.
                   Free CMA is 72900kB
                   Total reserve is 92268kB
                   Total free pages is 229828kB
                   Total file cache is 706092kB
                   Slab Reclaimable is 108216kB
                   Slab UnReclaimable is 422068kB
                   Total Slab is 530284kB
                   GFP mask is 0x6280c2
                   Kernel Memory Reclaimable is 41084kB
                   Free Swap 690460kB
                   queue len is 1 of max 1 reason:0x2 margin:16165
[142248.169577] oom_reaper: reaped process 29728 (AndroidUI), now anon-rss:0kB, file-rss:160kB, shmem-rss:820kB
[142248.220399] oom_reaper: reaped process 28697 (com.Slack), now anon-rss:0kB, file-rss:152kB, shmem-rss:3416kB
[142261.194268] lowmemorykiller: Killing 'a.fennec_fdroid' (30025), adj 900,
                   to free 25kB on behalf of 'kswapd-1:0' (102) because
                   cache 604272kB is below limit 945120kB for oom_score_adj 529
                   Free memory is 830024kB above reserved.
                   Free CMA is 72920kB
                   Total reserve is 92268kB
                   Total free pages is 286460kB
                   Total file cache is 684272kB
                   Slab Reclaimable is 107744kB
                   Slab UnReclaimable is 421896kB
                   Total Slab is 529640kB
                   GFP mask is 0x0
                   Kernel Memory Reclaimable is 21108kB
                   Free Swap 687140kB
                   queue len is 1 of max 1 reason:0x1 margin:30444
[142261.207562] oom_reaper: reaped process 30025 (a.fennec_fdroid), now anon-rss:0kB, file-rss:92kB, shmem-rss:116kB
[142262.683725] lowmemorykiller: Killing 'a.fennec_fdroid' (30200), adj 900,
                   to free 29kB on behalf of 'kswapd-1:0' (102) because
                   cache 653772kB is below limit 945120kB for oom_score_adj 529
                   Free memory is 738416kB above reserved.
                   Free CMA is 72924kB
                   Total reserve is 92268kB
                   Total free pages is 198104kB
                   Total file cache is 742164kB
                   Slab Reclaimable is 108088kB
                   Slab UnReclaimable is 421764kB
                   Total Slab is 529852kB
                   GFP mask is 0x0
                   Kernel Memory Reclaimable is 18620kB
                   Free Swap 686884kB
                   queue len is 1 of max 1 reason:0x1 margin:8228
[142262.701606] oom_reaper: reaped process 30200 (a.fennec_fdroid), now anon-rss:0kB, file-rss:0kB, shmem-rss:144kB
[142267.151109] lowmemorykiller: Killing 'AndroidUI' (30292), adj 900,
                   to free 32kB on behalf of 'kswapd-1:0' (102) because
                   cache 646052kB is below limit 945120kB for oom_score_adj 529
                   Free memory is 756040kB above reserved.
                   Free CMA is 72924kB
                   Total reserve is 92268kB
                   Total free pages is 164940kB
                   Total file cache is 735304kB
                   Slab Reclaimable is 108592kB
                   Slab UnReclaimable is 422432kB
                   Total Slab is 531024kB
                   GFP mask is 0x0
                   Kernel Memory Reclaimable is 69920kB
                   Free Swap 686372kB
                   queue len is 1 of max 1 reason:0x1 margin:-63
[142267.182397] oom_reaper: reaped process 30292 (AndroidUI), now anon-rss:0kB, file-rss:96kB, shmem-rss:480kB
[142270.285857] lowmemorykiller: Killing 'sailfish-qml' (29192), adj 103,
                   to free 201916kB on behalf of 'allocator-servi' (3333) because
                   cache 640788kB is below limit 760800kB for oom_score_adj 58
                   Free memory is 724820kB above reserved.
                   Free CMA is 72924kB
                   Total reserve is 92268kB
                   Total free pages is 186332kB
                   Total file cache is 729688kB
                   Slab Reclaimable is 108844kB
                   Slab UnReclaimable is 421884kB
                   Total Slab is 530728kB
                   GFP mask is 0x6280c2
                   Kernel Memory Reclaimable is 17308kB
                   Free Swap 686372kB
                   queue len is 1 of max 1 reason:0x2 margin:5285
[142270.330164] oom_reaper: reaped process 29192 (sailfish-qml), now anon-rss:0kB, file-rss:10016kB, shmem-rss:48kB

Then again, the amount of kills is expected to be non-zero, because mpris-proxy often needs a restart, Email vanishes right when I’m using it, the whole Android support is killed and automatically restarts etc. Just switching between Browser, WhatsApp and Slack is enouch to trigger a steady stream of kills.

Further notifications at this point:

The kernel oom killer didn’t take action
lowmemorykiller is trigger-happy
logmemorykiller can be configured
There seems to be (huge) differences between Android and AOSP-based operating systems (more RAM = more snipes?) (I lost the source in the swamp, sorry)

Here are my minfree values on my Xperia 10 III:

$ cat /sys/module/lowmemorykiller/parameters/minfree
167160,190200,213240,236280,356490 # n:o pages
$ getconf PAGESIZE
4096 # page size in bytes

Which correspond to roughly 650MB, 740MB, 830MB, 920MB and 1390MB. Those are the limits of free RAM (or usable, I’m not sure really) under which lmkd starts killing processes in different categories. Combining this with swappiness of

$ cat /proc/sys/vm/swappiness 
25

means that the system is likely to start sniping processes before swapping has a change to step in. I think. I have no memory management experience, but the swappiness sounds too low and the memory limits too strict. So, I’m going to adjust those values, and report back how it goes.

direc85 · 15 August 2022 17:43

Additional note: When Android app support starts, I get the following messages:

[147459.694751] init: Command 'chown root system /sys/module/lowmemorykiller/parameters/adj' action=boot (/init.rc:670) took 0ms and failed: lchown() failed: Read-only file system
[147459.694865] init: Command 'chmod 0664 /sys/module/lowmemorykiller/parameters/adj' action=boot (/init.rc:671) took 0ms and failed: fchmodat() failed: Read-only file system
[147459.694926] init: Command 'chown root system /sys/module/lowmemorykiller/parameters/minfree' action=boot (/init.rc:672) took 0ms and failed: lchown() failed: Read-only file system
[147459.695003] init: Command 'chmod 0664 /sys/module/lowmemorykiller/parameters/minfree' action=boot (/init.rc:673) took 0ms and failed: fchmodat() failed: Read-only file system

So there seems to be an lmkd in the alien dalvik side, too, and it would like to tinker with the setup. It cannot, which is good, I think.

direc85 · 15 August 2022 18:09

I also stumbled upon this nice little command:

$ zramctl 
NAME       ALGORITHM DISKSIZE   DATA COMPR TOTAL STREAMS MOUNTPOINT
/dev/zram0 lz4             1G 256,5M 62,9M 73,7M       8 [SWAP]

It tells me that it contains ~256MB of memory swapped, which takes ~63MB compressed, and the whole zram0 consumes ~73MB of memory. This seems really efficient, as from what I have found, typical compression ratio is between 2:1 and 3:1, this is around 4:1! (I wonder if zstd would be suitable for zram… Edit: Yes, but it may not be worth the performance hit.)

Maus · 15 August 2022 18:36

Thank you very much, @direc85 , for digging into this. After 18 months of using a “replacement” operating system, I plan to return to Sailfish, but the operating system randomly kills task, and if you’re “lucky” enough it’s lipstick. I do rely on some apps surviving this massacre and having 4 Go of RAM should make the device a lot more flexible than a J1. If this added flexibility comes with a huge loss of reliability, I really want my J1 back.
Enough lamenting: I hope your efforts will help to improve this!
On a system with 18 days of uptime, zramctl yields:

NAME       ALGORITHM DISKSIZE   DATA  COMPR  TOTAL STREAMS MOUNTPOINT
/dev/zram0 lz4             1G 998.5M 356.1M 373.7M       8 [SWAP]

I guess this will simply fill with growing uptime, and it makes sense to do so.

hackman238 · 15 August 2022 18:55

Another option here is to create a cronjob that alters the OOM score adjustment per process using the following on a periodic interval:

pgrep -f "processName" | while read PID; do echo "-1000" >/proc/$PID/oom_score_adj; done

I use this method to prevent the killing of lipstick, ofono and signal under any circumstances.

Another option, for processes which are daemons, one may alter the systemd unit (or craft one if it does not exist) to include the following under the Service stanza:

OOMScoreAdjust=-1000

This is the method I use to start and run s1p at all times and prevent any accidental killings.

Example: /etc/systemd/system/s1p.service

[Unit]
Description=S1P Service

Wants=network.target
After=syslog.target network-online.target

[Service]
Type=simple
User=root
OOMScoreAdjust=-1000
Environment="DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/100000/dbus/user_bus_socket"
ExecStart=/bin/su - defaultuser -c "/bin/bash /usr/share/harbour-s1p/sipBabysitter.sh"
ExecStop=/bin/su - root -c "/bin/ps aux | /bin/grep s1p | /bin/grep -v grep | /bin/awk '{print $1}' | /usr/bin/xargs /bin/kill -9"
Restart=always
RestartSec=10
KillMode=process

[Install]
WantedBy=multi-user.target`

Mined you it is possible to start any SFOs app this way- just create a wrapper using the following example and the service example above. It is critical to include the DBUS environment lines. Don’t ignore those. This babysitter script also implements hanging process cleanup on failure.

Example: /usr/share/harbour-s1p/sipBabysitter.sh

#!/bin/bash
export DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/100000/dbus/user_bus_socket
checkInterval=30

function StartProcesses {
    echo "s1p sipBabysitter - Starting main s1p processes..."
    /usr/bin/invoker -o --type=generic --single-instance s1p --qml harbour-s1p &
    echo "Waiting 30s before polling for s1p processes..."
    sleep 30
}

function KillProcesses {
    echo "s1p sipBabysitter - Doing process cleanup..."
    /bin/ps aux | /bin/grep s1p | /bin/grep -v grep | /bin/awk '{print $1}' | /usr/bin/xargs /bin/kill -9
}

function CheckProcesses {
    echo "s1p babySitter - Checking processess..."
    testSipServer=$(ps aux | grep "/usr/bin/s1p" | grep -v grep | wc -l)
    testLauncher=$(ps aux | grep "/usr/bin/invoker" | grep "s1p --qml harbour-s1p" | grep -v grep | wc -l)
    testUi=$(ps aux | grep "sailfish-qml harbour-s1p" | grep -v grep | wc -l)

    if [ $testSipServer -eq 1 ] && [ $testLauncher -eq 1 ] && [ $testUi -eq 1 ]
    then
        return 0
    else
        return 1
    fi
}

StartProcesses
CheckProcesses
resValue=$?
while [ $resValue -eq 0 ]
do
  echo "s1p babySitter - Process Check result: OK"
  sleep $checkInterval
  CheckProcesses
  resValue=$?
done

echo "s1p babySitter - Process Check result: Failed"
KillProcesses
exit 1

direc85 · 15 August 2022 21:58

@Maus, I can assure you; after I modified the values, my X10III feels like it actually has 6GB of RAM! If my phone felt like a beast before (even when compared with Xperia 10 II), now it’s a monster!

@hackman238 That is both very neat and unfortunately hacky at the same time…! It’s great that the oom_score_adj can be easily set per-process, but it really shouldn’t have to be done via a shell script. I tried to search for a bit, but I didn’t find any obvious way to set a desired value at startup.

However; I found this question which reveals that a process can write value to /proc/self/oom_score_adj, so that is something that the binary can do by itself, which is nice! Another option is to have a dedicated adjust daemon with a list of processes and adjustments, I guess.

Anyway, after a few hours of testing, with a stable phone still, here are the current values – I simply halved each number and doubled the swappiness.

$ cat /sys/module/lowmemorykiller/parameters/minfree
83580,95100,106620,118140,178245
$ cat /proc/sys/vm/swappiness
50

Note: These are from Xperia 10 III with 6GB of RAM! The propotion of the numbers relative to the physical memory are as follows:

Old: 10.6% 12.1% 13.6% 15.0% 22.7%
New:  5.3%  6.0%  6.8%  7.5% 11.3%

Old: 653.0MiB 743.0MiB 833.0MiB 923.0MiB 1392.5MiB
New: 326.5MiB 372.5MiB 416.5MiB 461.5MiB  696.3MiB

(I think the values could be a lot lower still, the tinkering continues…)

And the current memory usage:

$ cat /proc/meminfo 
MemTotal:        5640112 kB
MemFree:          510008 kB
MemAvailable:    1085776 kB
Buffers:           64880 kB
Cached:           602544 kB
SwapCached:        14832 kB
Active:          1435476 kB
Inactive:        1527836 kB
Active(anon):    1153812 kB
Inactive(anon):  1227808 kB
Active(file):     281664 kB
Inactive(file):   300028 kB
Unevictable:       13148 kB
Mlocked:           13148 kB
SwapTotal:       1048572 kB
SwapFree:         730044 kB
[...]
$ zswapctl
NAME       ALGORITHM DISKSIZE   DATA COMPR TOTAL STREAMS MOUNTPOINT
/dev/zram0 lz4             1G 307,8M 77,5M 81,8M       8 [SWAP]

I’m simply blown away. The swap is chilling there with third of the capacity used!

I’m currently running Browser, WhatsApp, Slack, Hydrogen, Whisperfish, “Firefox”, HERE WeGo, Deezer, Gallery and Email, all at once, and none of them gets killed when I rapidly switch between them (I’ve been doing this for a few minutes now, going strong)! The fingerprint sensor still works as it should, and so does my Bluetooth headphones’ media controls (aka. mpris-proxy hasn’t been killed either)!

I still have ~179MB free RAM and ~1100MB available (used by cache, free-able anytime) RAM, this is nothing short of a breakthrough! And there are zero processes killed since I made the change above.

For comparison, here’re the numbers from Jolla Phone:

$ cat /sys/module/lowmemorykiller/parameters/minfree 
4096,6144,8192,12288,20000
$ zramctl 
NAME       ALGORITHM DISKSIZE  DATA COMPR TOTAL STREAMS MOUNTPOINT
/dev/zram1              52,6M  5,5M  1,9M  2,9M         [SWAP]
/dev/zram0              52,6M  5,4M  1,9M  2,9M         [SWAP]

 2.0%  3.0%  4.0%  5.9%  9.7%
16MiB 24MiB 32MiB 48MiB 78.1MiB

So my adjusted (halved) values above are still higher in relative terms to the total RAM size, which indicates to me that the default values in Xperia 10 III actually are stupendously large.

fingus · 16 August 2022 08:05

How to Adjust those Values for the Sony Xperia X Compact?

The Device has 3GB RAM, which should be enough to run my Apps (Fernschreiber, Jollawebbrowser, E-Mail)

direc85 · 16 August 2022 10:53

First determine the current values and determine if they are “unreasonably high”. This is from an Xperia 10 with 3GB (or 4GB, I dunno) of RAM:

cat /sys/module/lowmemorykiller/parameters/minfree
15360,19200,23040,26880,34415,43737

Looking at the highest value at the end, 43737, turn it into megabytes: 43737/4*1024=170.8MB. That seems reasonable, considering the max 3-4GB or RAM. With my X10III the max value of 356490 turned into 1392MB, which is not fine, even with the 6GB. (What it means, is that lmkd starts killing processes first when free memory dips below 1392MB.)

Then again, my Xperia XA2 Ultra has even lower numbers:

1536,2048,4096,16384 (but there’s only four parameters, not six??)

Okay, how about my Xperia X? It has the same four values. Okay. Perhaps that has something to do with armv7hl vs. aarch64? I literally do not know at this point. Both values are totally fine (quite small actually).

If your numbers needs adjustment, I suggest first closing all apps and Android support to have maximum amount of free RAM, and then setting the values (with root privileges) like this, assuming you have six values (values taken from the Xperia 10):

echo -n “15360,19200,23040,26880,34415,43737” > /sys/module/lowmemorykiller/parameters/minfree

As always, becareful with the values, you’re doing this at your own risk.

Edit: I’m testing the values above with my Xperia 10 III now, I was able to be set them correctly at least.

fingus · 16 August 2022 11:57

,—
| Sailfish OS 4.4.0.68 (Vanha Rauma)
'—
[defaultuser@XperiaXCompact ~]$ cat /sys/module/lowmemorykiller/parameters/minfree
1536,2048,4096,16384

At least only 4 Values.

Specially this two Apps suffer restarts from time to time:

Fernschreiber and Jolla WebBrowser.

direc85 · 16 August 2022 12:05

Those do seem almost unreasonably small… Even the largest portion is only 2.1% of the total available RAM! Try increasing the values instead:

3072,4096,8192,32768

That’s 12MiB, 16MiB, 32MiB, 128MiB and 0.4%, 0.5%, 1.0%, 4.2% respectively.

I suggest also kicking swappines to 50 - assuming you have zram around 512MiB…1GiB (check it by running zramctl as root)

fingus · 16 August 2022 12:07

NAME ALGORITHM DISKSIZE DATA COMPR TOTAL STREAMS MOUNTPOINT
/dev/zram3 lzo 139,8M 636K 279,9K 560K 1 [SWAP]
/dev/zram2 lzo 139,8M 652K 287,8K 576K 1 [SWAP]
/dev/zram1 lzo 139,8M 644K 308,6K 600K 1 [SWAP]
/dev/zram0 lzo 139,8M 624K 298,6K 560K 1 [SWAP]

direc85 · 16 August 2022 12:08

That’s around half a giga, so swappiness of 50 should be fine.

fingus · 16 August 2022 12:10

[root@XperiaXCompact defaultuser]# cat /proc/sys/vm/swappiness
60

So i set it to 50 like you suggest.

Is a restart mandatory after changing these Values?

thigg · 16 August 2022 12:15

According to your first post, the sizes are in pages. Thus you need to make sure, that pagesize is the same when you compare those values between phones

direc85 · 16 August 2022 12:26

That’s the default value; I think that’s a bit too aggressive, try 40-50 instead…

I started getting Browser crashes (they weren’t lkmd kills) so I’m using thirds of the original values (X10III w/6BG):

55720,63400,71080,78760,118830

Indeed. All the four devices I have at hand have the same 4096 bytes page size (even my desktop PC has it) but other sizes do exist, so it has to be checked.

karry · 16 August 2022 12:32

Hi, you may be interested in my older blog post for this topic: Sailfish OS and memory :: karry.cz

just few corrections:

Linux kernel contains standard OOM killer, and Sailfish OS adds even non-standard lowmemory killer module. This is the guy that is causing troubles.
lmkd is android user-space lowmemory killer replacement. On Sailfish OS, it is running in container and affects just android applications.

My conclusion is that memory management in Sailfish is broken and in-kernel lowmemory killer should be replaced by some smarter user-space alternative. lmkd, systemd or SFOS specific implementation…

thigg · 16 August 2022 12:39

Combining this with swappiness of

$ cat /proc/sys/vm/swappiness
25

means that the system is likely to start sniping processes before swapping has a change to step in.

After reading this I am quite sure this is wrong. As I understand it, you could notice a performance impact by changing the swappiness, but it shouldnt influence the oom behavior in any way.

Please edit your first post if you come to the same conclusion. (or share your knowledge with me to convince me otherwise

However, I really appreciate that you are digging into this.

JoOppen · 16 August 2022 16:22

Thank you for tackling this problem! I would be great if Jolla would adopt your solution in the next release (trusting that your approach is going in the right direction)

direc85 · 16 August 2022 21:06

Thank you for the suggestions and corrections, @karry and @thigg! I’ll review and update the first post tomorrow.

About lowmemorykiller:

That was quite an interesting read, karry! It looks like you tackled this before than me, and with more technical approach, too!

It is indeed an Android user-space daemon (that much I did get right) but I assumed it was running on the Sailfish OS side covering the whole system, because of SFOS is running on top of AOSP and that made all the sense to me. Should have checked though. The process lmkd is running only when the Android app support is running, so that indeed is the case.

That means my impression of setting lower values to the minfree file affecting e.g. Browser is 100% placebo. I guess the increased overall responsiveness just carried over. Edit: This also means Email and Browser simply vanishing on me is not because of oom but another bugs altogether. Oh dear.

However, this means that there’s no SFOS side user-space daemon to handle oom. (Or is there? I found nothing when I searched…) Having two oom daemons monitoring the same RAM space doesn’t really sound welcoming either. Perhaps something simple like only using earlyoom SFOS-side with system application oom score adjustment could work? The in-kernel oom killer doesn’t get a lot of praises, from what I’ve read…

About swappiness:

Indeed, I didn’t really understand vm.swappiness; it doesn’t mean what I think it did. After reading the excellent blog post, I think 25 is a good value for SFOS configuration. I’m going to keep using 50 still, as I simply prefer swapping over killed processes, and changing swappiness from 25 to 50 isn’t that drastic. The value goes up to 200 after all. There’s a lot of inactive stuff to page out, specially with 6GiB of applications loaded.

Then again, in another thread there’s someone with a swappiness of 60, but it’s a community port, and I guess swappiness value just wasn’t ever checked or considered.

JacekJagosz · 16 August 2022 22:23

Wow, amazing read! Thank for writing something so informative, I hope someone from Jolla will look at it.
Meanwhile I wonder if we could tweak oom_score_adj in any way? Or treshold at whick mce reports?
I wonder how many apps actually try to use mce’s memory pressure reporting? Does browser do it? It definitely should.
Also wouldn’t increasing swappiness actually help in this situation? With the same apps running and number seen by mce, there is more free RAM left, so apps should be killed a bit later. Also more chance for MCE to report pressure.