Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

nohang-desktop stopped working? #142

Open
dim-geo opened this issue Dec 23, 2024 · 15 comments
Open

nohang-desktop stopped working? #142

dim-geo opened this issue Dec 23, 2024 · 15 comments

Comments

@dim-geo
Copy link

dim-geo commented Dec 23, 2024

Hi, I use cachy-os kernel patches & zram and I have noticed that nohang-desktop is not protecting from OOM anymore.
kswapd0 is triggering oom-killer.

How can I debug this further? Can I change some limits in nohang-desktop to force nohang to intervene earlier?

Logs:

Dec 23 12:11:20 gentoo nohang-desktop[81157]: PSI:   5.76 | MemAvail:  9578 M, 40.8 % | SwapFree:  528 M,   9.0 % | MemUsedZram:   479 M,  2.0 % | dMem:   -36 M/s
Dec 23 12:12:21 gentoo nohang-desktop[81157]: PSI:   2.82 | MemAvail:  9880 M, 42.1 % | SwapFree:  951 M,  16.2 % | MemUsedZram:   480 M,  2.0 % | dMem:    12 M/s
...
Dec 23 12:13:22 gentoo nohang-desktop[81157]: PSI:   7.95 | MemAvail:  9149 M, 39.0 % | SwapFree:  654 M,  11.1 % | MemUsedZram:   479 M,  2.0 % | dMem:   -17 M/s
Dec 23 12:13:33 gentoo kernel: kswapd0 invoked oom-killer: gfp_mask=0xcc0(GFP_KERNEL), order=0, oom_score_adj=0
Dec 23 12:13:33 gentoo kernel: CPU: 0 UID: 0 PID: 74 Comm: kswapd0 Tainted: G S      W  OE      6.12.6-cachyos-6.12.6-x86_64 #1
Dec 23 12:13:33 gentoo kernel: Tainted: [S]=CPU_OUT_OF_SPEC, [W]=WARN, [O]=OOT_MODULE, [E]=UNSIGNED_MODULE
Dec 23 12:13:33 gentoo kernel: Hardware name:
Dec 23 12:13:33 gentoo kernel: Sched_ext: flash (enabled+all), task: runnable_at=-1ms
Dec 23 12:13:33 gentoo kernel: Call Trace:
Dec 23 12:13:33 gentoo kernel:  <TASK>
Dec 23 12:13:33 gentoo kernel:  dump_stack_lvl+0x56/0x80
Dec 23 12:13:33 gentoo kernel:  dump_header+0x50/0x36d
Dec 23 12:13:33 gentoo kernel:  ? bpf_task_storage_get+0xec/0x100
Dec 23 12:13:33 gentoo kernel:  ? ktime_get_mono_fast_ns+0x30/0xb0
Dec 23 12:13:33 gentoo kernel:  ? ___ratelimit+0x99/0xf0
Dec 23 12:13:33 gentoo kernel:  oom_kill_process.cold+0x8/0x8a
Dec 23 12:13:33 gentoo kernel:  out_of_memory+0x102/0x410
Dec 23 12:13:33 gentoo kernel:  balance_pgdat+0x778/0xb30
Dec 23 12:13:33 gentoo kernel:  kswapd+0x1cd/0x350
Dec 23 12:13:33 gentoo kernel:  ? complete+0x80/0x80
Dec 23 12:13:33 gentoo kernel:  ? balance_pgdat+0xb30/0xb30
Dec 23 12:13:33 gentoo kernel:  kthread+0xcb/0x100
Dec 23 12:13:33 gentoo kernel:  ? kthread_park+0x80/0x80
Dec 23 12:13:33 gentoo kernel:  ret_from_fork+0x2d/0x50
Dec 23 12:13:33 gentoo kernel:  ? kthread_park+0x80/0x80
Dec 23 12:13:33 gentoo kernel:  ret_from_fork_asm+0x11/0x20
Dec 23 12:13:33 gentoo kernel:  </TASK>
Dec 23 12:13:33 gentoo kernel: Mem-Info:
Dec 23 12:13:33 gentoo kernel: active_anon:1399496 inactive_anon:1800837 isolated_anon:0
                                   active_file:364556 inactive_file:538027 isolated_file:59
                                   unevictable:7488 dirty:1723 writeback:0
                                   slab_reclaimable:65189 slab_unreclaimable:82292
                                   mapped:134366 shmem:5761 pagetables:22881
                                   sec_pagetables:5215 bounce:0
                                   kernel_misc_reclaimable:0
                                   free:1303866 free_pcp:1856 free_cma:0
Dec 23 12:13:33 gentoo kernel: Node 0 active_anon:5597984kB inactive_anon:7203348kB active_file:1458224kB inactive_file:2152108kB unevictable:29952kB isolated(anon):0kB isolated(file):236kB mapped:537464kB dirty:6892kB writeback:0kB shmem:23044kB shmem_thp:0kB shmem_pmdmapped:0kB anon_thp:9814016kB writeback_tmp:0kB kernel_stack:23036kB pagetables:91524kB sec_pagetables:20860kB all_unreclaimable? no
Dec 23 12:13:33 gentoo kernel: DMA free:1024kB boost:0kB min:40kB low:52kB high:64kB reserved_highatomic:0KB active_anon:0kB inactive_anon:10240kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15984kB managed:15360kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Dec 23 12:13:33 gentoo kernel: lowmem_reserve[]: 0 2683 23474 0 0
Dec 23 12:13:33 gentoo kernel: DMA32 free:86388kB boost:0kB min:7672kB low:10400kB high:13128kB reserved_highatomic:0KB active_anon:184136kB inactive_anon:2449644kB active_file:1124kB inactive_file:868kB unevictable:0kB writepending:0kB present:2813388kB managed:2747852kB mlocked:0kB bounce:0kB free_pcp:2696kB local_pcp:648kB free_cma:0kB
Dec 23 12:13:33 gentoo kernel: lowmem_reserve[]: 0 0 20790 0 0
Dec 23 12:13:33 gentoo kernel: Normal free:5128052kB boost:0kB min:59864kB low:81148kB high:102432kB reserved_highatomic:2048KB active_anon:5413384kB inactive_anon:4743680kB active_file:1456068kB inactive_file:2151516kB unevictable:29936kB writepending:6660kB present:21772288kB managed:21289640kB mlocked:29628kB bounce:0kB free_pcp:4720kB local_pcp:1316kB free_cma:0kB
Dec 23 12:13:33 gentoo kernel: lowmem_reserve[]: 0 0 0 0 0
Dec 23 12:13:33 gentoo kernel: DMA: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 1*1024kB (U) 0*2048kB 0*4096kB = 1024kB
Dec 23 12:13:33 gentoo kernel: DMA32: 454*4kB (UME) 331*8kB (UME) 224*16kB (UME) 162*32kB (UME) 109*64kB (UME) 69*128kB (UME) 53*256kB (UME) 42*512kB (UME) 22*1024kB (UM) 0*2048kB 0*4096kB = 86640kB
Dec 23 12:13:33 gentoo kernel: Normal: 22142*4kB (UE) 38717*8kB (UE) 68941*16kB (UMEH) 28856*32kB (UMEH) 10626*64kB (UME) 4913*128kB (UMEH) 2383*256kB (UME) 981*512kB (UMEH) 275*1024kB (UMEH) 0*2048kB 0*4096kB = 5127600kB
Dec 23 12:13:33 gentoo kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Dec 23 12:13:33 gentoo kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Dec 23 12:13:33 gentoo kernel: 913829 total pagecache pages
Dec 23 12:13:33 gentoo kernel: 2066 pages in swap cache
Dec 23 12:13:33 gentoo kernel: Free swap  = 324kB
Dec 23 12:13:33 gentoo kernel: Total swap = 6013180kB
Dec 23 12:13:33 gentoo kernel: 6150415 pages RAM
Dec 23 12:13:33 gentoo kernel: 0 pages HighMem/MovableOnly
Dec 23 12:13:33 gentoo kernel: 137202 pages reserved
Dec 23 12:13:33 gentoo kernel: 0 pages cma reserved
Dec 23 12:13:33 gentoo kernel: 0 pages hwpoisoned
Dec 23 12:13:33 gentoo kernel: Tasks state (memory values in pages):
Dec 23 12:13:33 gentoo kernel: [  pid  ]   uid  tgid total_vm      rss rss_anon rss_file rss_shmem pgtables_bytes swapents oom_score_adj name
Dec 23 12:13:33 gentoo kernel: [   2898]     0  2898    45425    11955        0    11955         0   409600      480          -250 systemd-journal
Dec 23 12:13:33 gentoo kernel: [   2934]     0  2934     8644     1248        0     1248         0    94208     1184         -1000 systemd-udevd
Dec 23 12:13:33 gentoo kernel: [   4630]   129  4630     4705     1374        0     1374         0    81920      352             0 systemd-network
Dec 23 12:13:33 gentoo kernel: [   4760]   101  4760     2416      736        0      736         0    65536      608          -900 dbus-daemon
Dec 23 12:13:33 gentoo kernel: [   4761]     0  4761     2001      839        0      839         0    57344      128             0 bash
Dec 23 12:13:33 gentoo kernel: [   4768]     0  4768     2001      454        0      454         0    53248      160             0 bash
Dec 23 12:13:33 gentoo kernel: [   4773]     0  4773    18967     2582      480     2102         0    73728        0             0 ntpd
Dec 23 12:13:33 gentoo kernel: [   4780]   126  4780     5056      613        0      613         0    53248       64             0 rtkit-daemon
Dec 23 12:13:33 gentoo kernel: [   4781]     0  4781   103335      492        0      492         0   102400      128             0 scx_loader
Dec 23 12:13:33 gentoo kernel: [   4783]     0  4783     2121      990        0      990         0    53248      384             0 smartd
Dec 23 12:13:33 gentoo kernel: [   4787]     0  4787     4607     1504       64     1440         0    77824      832             0 snmpd
Dec 23 12:13:33 gentoo kernel: [   4801]     0  4801     3655     1227        0     1227         0    73728      384             0 systemd-logind
Dec 23 12:13:33 gentoo kernel: [   4802]     0  4802    10435      313        0      313         0    69632      192             0 sensord
Dec 23 12:13:33 gentoo kernel: [   4804]     0  4804     3452     1121        0     1121         0    73728      256             0 systemd-machine
Dec 23 12:13:33 gentoo kernel: [   4808]     0  4808      591      363        0      363         0    49152        0             0 lowlatency
Dec 23 12:13:33 gentoo kernel: [   4810]     0  4810    41452     1215       96     1119         0    73728      544          -999 ananicy-cpp
Dec 23 12:13:33 gentoo kernel: [   4824]     0  4824      618      259        0      259         0    45056        0             0 netatopd
Dec 23 12:13:33 gentoo kernel: [   4831]    84  4831     9029      988       32      956         0    69632      160             0 upsd
Dec 23 12:13:33 gentoo kernel: [   4852]     0  4852    54739      538       27      511         0    94208     2272             0 scx_flash
Dec 23 12:13:33 gentoo kernel: [   4856]     0  4856    88836     4069        0     4069         0   200704     6880             0 python3
Dec 23 12:13:33 gentoo kernel: [   4881]     0  4881    36929     3777        0     3777         0   147456      544             0 sddm
Dec 23 12:13:33 gentoo kernel: [   4884]     0  4884     2346      652        0      652         0    57344      160             0 upsmon
Dec 23 12:13:33 gentoo kernel: [   4976]    84  4976     2942     1425        8     1417         0    61440      256             0 upsmon
Dec 23 12:13:33 gentoo kernel: [   5117] 65534  5117     1424      328        1      327         0    53248       96             0 dnsmasq
Dec 23 12:13:33 gentoo kernel: [   5118]     0  5118     1424      296       33      263         0    53248       32             0 dnsmasq
Dec 23 12:13:33 gentoo kernel: [   5149] 65534  5149     1424      313        1      312         0    57344       64             0 dnsmasq
Dec 23 12:13:33 gentoo kernel: [   5150]     0  5150     1424      129        1      128         0    57344       64             0 dnsmasq
Dec 23 12:13:33 gentoo kernel: [   5153]     0  5153    15066      698        0      698         0   155648     6624             0 python3
Dec 23 12:13:33 gentoo kernel: [   5345]     0  5345    17298     1801        0     1801         0   135168      608             0 nmbd
Dec 23 12:13:33 gentoo kernel: [   5347]   122  5347    95334     1533        0     1533         0   102400      672             0 polkitd
Dec 23 12:13:33 gentoo kernel: [   5350]     0  5350    18475     1894        0     1883        11   143360      640             0 winbindd
Dec 23 12:13:33 gentoo kernel: [   5365]     0  5365    18521     1284        0     1273        11   139264      704             0 wb[GENTOO]
Dec 23 12:13:33 gentoo kernel: [   5366]     0  5366    19556     1901        0     1871        30   151552      672             0 smbd
Dec 23 12:13:33 gentoo kernel: [   5370]     0  5370    19088      503        2      501         0   122880      608             0 smbd-notifyd
Dec 23 12:13:33 gentoo kernel: [   5371]     0  5371    19088      418        0      418         0   118784      672             0 smbd-cleanupd
Dec 23 12:13:33 gentoo kernel: [   5383]     0  5383    18475      648       30      618         0   122880      576             0 wb-idmap
Dec 23 12:13:33 gentoo kernel: [   5567]    84  5567     1236      384        1      383         0    49152       32             0 blazer_ser
Dec 23 12:13:33 gentoo kernel: [   6142]     0  6142    77246     1263        0     1263         0   110592      672             0 upowerd
Dec 23 12:13:33 gentoo kernel: [   6190]     0  6190   116354     1794        0     1794         0   139264      928             0 udisksd
Dec 23 12:13:33 gentoo kernel: [   6202]     0  6202    77027     1281        0     1281         0   102400      160             0 power-profiles-
Dec 23 12:13:33 gentoo kernel: [   8644]     0  8644   426846     2949        0     2949         0   380928     7584             0 libvirtd
Dec 23 12:13:33 gentoo kernel: [   8981]     0  8981    17277     2498        0     2498         0   126976      448          -900 virtlogd
Dec 23 12:13:33 gentoo kernel: [  10361]     0 10361   283303    19651     2357    11985      5309   667648     8555             0 X
Dec 23 12:13:33 gentoo kernel: [  10391]  1000 10391  1893251  1068987  1053385    15597         5 10465280    11993             0 qemu-system-x86
Dec 23 12:13:33 gentoo kernel: [  10501]     0 10501    19753     1255        0     1255         0   155648      704             0 smbd[192.168.12
Dec 23 12:13:33 gentoo kernel: [  10692]     0 10692    18973     3639        0     3639         0   143360      512             0 sddm-helper
Dec 23 12:13:33 gentoo kernel: [  10698]  1000 10698     4363     1423        0     1423         0    73728      768           100 systemd
Dec 23 12:13:33 gentoo kernel: [  10700]  1000 10700     4185      499        6      493         0    65536      320           100 (sd-pam)
Dec 23 12:13:33 gentoo kernel: [  10716]  1000 10716    56769     5837     2765     2458       614   225280     3115           200 pipewire
Dec 23 12:13:33 gentoo kernel: [  10717]  1000 10717   313067      165       20      145         0   147456     2240           200 syncthing
Dec 23 12:13:33 gentoo kernel: [  10718]  1000 10718   150037     3310       27     3283         0   282624     2391           200 wireplumber
Dec 23 12:13:33 gentoo kernel: [  10720]  1000 10720    44355     1448        0     1421        27   122880     3220           200 pipewire-pulse
Dec 23 12:13:33 gentoo kernel: [  10730]  1000 10730   282600    11868       60    11808         0   589824     3680             0 kwalletd6
Dec 23 12:13:33 gentoo kernel: [  10735]  1000 10735     2003      899      128      771         0    53248      480           200 dbus-daemon
Dec 23 12:13:33 gentoo kernel: [  10742]  1000 10742    53146     4349        0     4349         0   229376     1120             0 startplasma-x11
Dec 23 12:13:33 gentoo kernel: [  10749]  1000 10749   330348     6593     3805     2788         0   282624     5481           200 syncthing
Dec 23 12:13:33 gentoo kernel: [  10839]  1000 10839   269925    11049        0    11049         0   524288     3296           200 kglobalacceld
Dec 23 12:13:33 gentoo kernel: [  10846]  1000 10846   269067     1625        0     1625         0   196608      576           200 xdg-desktop-por
Dec 23 12:13:33 gentoo kernel: [  10853]  1000 10853   132849     1241        0     1241         0   135168      192           200 xdg-document-po
Dec 23 12:13:33 gentoo kernel: [  10860]  1000 10860    76908     1214        0     1214         0   102400      128           200 xdg-permission-
Dec 23 12:13:33 gentoo kernel: [  10870]  1000 10870      597      450        0      450         0    49152        0           200 fusermount3
Dec 23 12:13:33 gentoo kernel: [  10879]  1000 10879   281566    11830        0    11830         0   585728     3680           200 ksmserver
Dec 23 12:13:33 gentoo kernel: [  10881]  1000 10881   450453    13672       68    13604         0   802816     8864           200 kded6
Dec 23 12:13:33 gentoo kernel: [  10882]  1000 10882   547759    26390     8172    18218         0  1097728    10592           200 kwin_x11
Dec 23 12:13:33 gentoo kernel: [  10940]  1000 10940   824364    77152    46116    31036         0  2265088    72640           200 plasmashell
Dec 23 12:13:33 gentoo kernel: [  10955]  1000 10955    57434     1027        0     1027         0    86016      256           200 dconf-service
Dec 23 12:13:33 gentoo kernel: [  10987]  1000 10987   294306    11522        0    11522         0   540672     3360           200 kactivitymanage
Dec 23 12:13:33 gentoo kernel: [  10989]  1000 10989   262478    10200       38    10162         0   466944     2400           200 gmenudbusmenupr
Dec 23 12:13:33 gentoo kernel: [  10990]  1000 10990   281625    11660       44    11616         0   589824     3648           200 kaccess
Dec 23 12:13:33 gentoo kernel: [  10991]  1000 10991   281318    11948        0    11948         0   589824     3392           200 polkit-kde-auth
Dec 23 12:13:33 gentoo kernel: [  10992]  1000 10992   335590    11941        0    11941         0   606208     3264           200 org_kde_powerde
Dec 23 12:13:33 gentoo kernel: [  10993]  1000 10993   282106    11759        0    11759         0   577536     3776           200 xdg-desktop-por
Dec 23 12:13:33 gentoo kernel: [  10994]  1000 10994   262311    10159        0    10159         0   471040     2400           200 xembedsniproxy
Dec 23 12:13:33 gentoo kernel: [  11053]  1000 11053    41652     4677      608     4069         0   184320      960           200 ksystemstats
Dec 23 12:13:33 gentoo kernel: [  11102]  1000 11102     1865      910        0      910         0    61440       96           200 xsettingsd
Dec 23 12:13:33 gentoo kernel: [  11105]  1000 11105    55177     3615        0     3615         0   155648      640           200 kscreen_backend
Dec 23 12:13:33 gentoo kernel: [  11200]  1000 11200   266412     9388        5     9383         0   495616     2592           200 xdg-desktop-por
Dec 23 12:13:33 gentoo kernel: [  11202]  1000 11202   430159    14951     3775    11145        31   798720     5312           200 openrgb
Dec 23 12:13:33 gentoo kernel: [  11216]  1000 11216   282399    11856        0    11856         0   593920     3392           200 kclockd
Dec 23 12:13:33 gentoo kernel: [  11217]  1000 11217   309603    14917       88    14829         0   634880     5248           200 kdeconnectd
Dec 23 12:13:33 gentoo kernel: [  11244]  1000 11244    77058     1252        0     1252         0   110592      128           200 agent
Dec 23 12:13:33 gentoo kernel: [  11249]  1000 11249   375370    12313       34    12279         0   745472     5886           200 kalendarac
Dec 23 12:13:33 gentoo kernel: [  11299]  1000 11299    77700     1179        0     1179         0   110592      192           200 gvfsd
Dec 23 12:13:33 gentoo kernel: [  11324]  1000 11324    94606     1207        0     1207         0   102400      160           200 at-spi-bus-laun
Dec 23 12:13:33 gentoo kernel: [  11333]  1000 11333     1656      709        0      709         0    53248       96           200 dbus-daemon
Dec 23 12:13:33 gentoo kernel: [  11335]  1000 11335    58124     1403        0     1403         0    86016      192           200 at-spi2-registr
Dec 23 12:13:33 gentoo kernel: [  11548]     0 11548    88836     1563       23     1540         0   196608     6816             0 python3
Dec 23 12:13:33 gentoo kernel: [  11631]  1000 11631   663007    33148     4295    28853         0   962560    12800           200 looking-glass-c
Dec 23 12:13:33 gentoo kernel: [  12134]  1000 12134   281776    11896       66    11830         0   589824     3680           200 baloorunner
Dec 23 12:13:33 gentoo kernel: [  12792]  1000 12792   368691    25035     1848    17897      5290   786432     7449           200 konsole
Dec 23 12:13:33 gentoo kernel: [  12820]  1000 12820    33301     3269     1729     1540         0   167936     5664           200 python2
Dec 23 12:13:33 gentoo kernel: [  17378]  1000 17378  5210480    44311     6430    37881         0  3469312    89572           200 firefox-bin
Dec 23 12:13:33 gentoo kernel: [  17463]  1000 17463    52527     4224        0     4206        18   217088     2144           200 Socket Process
Dec 23 12:13:33 gentoo kernel: [  17506]  1000 17506   633751    16618      156    16419        43   946176    11086           167 Privileged Cont
Dec 23 12:13:33 gentoo kernel: [  17565]  1000 17565   649542    17652      597    17055         0  1150976    29432           167 Isolated Web Co
Dec 23 12:13:33 gentoo kernel: [  17572]  1000 17572   649165    17264      307    16950         7  1114112    20992           167 Isolated Web Co
Dec 23 12:13:33 gentoo kernel: [  17685]  1000 17685  4992737    24022     3241    20781         0  2707456    61220           100 WebExtensions
Dec 23 12:13:33 gentoo kernel: [  17716]  1000 17716    86461     5394        0     5394         0   389120     4384           200 Utility Process
Dec 23 12:13:33 gentoo kernel: [  18183]  1000 18183   636513    16997      454    16543         0   868352    12192           167 Isolated Web Co
Dec 23 12:13:33 gentoo kernel: [  18187]  1000 18187   635109    16764      309    16455         0   839680    12224           167 Isolated Web Co
Dec 23 12:13:33 gentoo kernel: [  18822]  1000 18822   633958    17381     1238    16143         0   884736     9291           167 Isolated Web Co
Dec 23 12:13:33 gentoo kernel: [  19265]  1000 19265   637918    16578      126    16439        13   937984    15904           167 Isolated Web Co
Dec 23 12:13:33 gentoo kernel: [  19976]   130 19976     5914     2234      192     2042         0    90112     1102             0 systemd-resolve
Dec 23 12:13:33 gentoo kernel: [  20391]  1000 20391     1935      802       32      770         0    53248       64             0 sh
Dec 23 12:13:33 gentoo kernel: [  20394]  1000 20394 68773504    10777     2573     8196         8   974848    19264             0 jackett
Dec 23 12:13:33 gentoo kernel: [  21269]  1000 21269   651007    21277     2833    18444         0  1105920    26381           100 Isolated Web Co
Dec 23 12:13:33 gentoo kernel: [  35655]  1000 35655   755983    27376     9785    17591         0  2920448    64486           167 Isolated Web Co
Dec 23 12:13:33 gentoo kernel: [  37314]  1000 37314   624910    15046      293    14718        35   770048     9014           167 Isolated Web Co
Dec 23 12:13:33 gentoo kernel: [  37415]  1000 37415   101712     6086       46     6015        25   471040     2921           200 RDD Process
Dec 23 12:13:33 gentoo kernel: [  37663]  1000 37663   664142    17055      372    16675         8  1347584    20205           167 Isolated Web Co
Dec 23 12:13:33 gentoo kernel: [  38274]  1000 38274   658276    19628      814    18802        12  1220608    30798           167 Isolated Web Co
Dec 23 12:13:33 gentoo kernel: [  38849]  1000 38849   670188    20306     1951    18314        41  1490944    23378           167 Isolated Web Co
Dec 23 12:13:33 gentoo kernel: [  38991]  1000 38991   656068    18542      441    18101         0  1204224    16354           167 Isolated Web Co
Dec 23 12:13:33 gentoo kernel: [  39893]  1000 39893   642394    19106      352    18732        22   995328    16110           167 Isolated Web Co
Dec 23 12:13:33 gentoo kernel: [  44248]  1000 44248   648367    18586      393    18193         0  1105920    14700           167 Isolated Web Co
Dec 23 12:13:33 gentoo kernel: [  58670]  1000 58670   655492    29989     5470    24519         0  1232896    14050           167 Isolated Web Co
Dec 23 12:13:33 gentoo kernel: [  80156]  1000 80156   327963    20244      174    20070         0   811008     8320           200 yakuake
Dec 23 12:13:33 gentoo kernel: [  80971]  1000 80971     2750      919       16      903         0    69632      864           200 bash
Dec 23 12:13:33 gentoo kernel: [  81001]  1000 81001     2749      912        0      912         0    65536      864           200 bash
Dec 23 12:13:33 gentoo kernel: [  81024]  1000 81024     2066      758        0      758         0    61440       96           200 su
Dec 23 12:13:33 gentoo kernel: [  81041]     0 81041     3342      903        0      903         0    69632     1472           200 bash
Dec 23 12:13:33 gentoo kernel: [  81157]     0 81157     6449     4910     3248     1662         0    98304        0             0 python3
Dec 23 12:13:33 gentoo kernel: [  81176]     0 81176   151825     3901      504     3397         0  1253376   145184           200 emerge
Dec 23 12:13:33 gentoo kernel: [  81223]  1000 81223     2749      922        0      922         0    61440      864           200 bash
Dec 23 12:13:33 gentoo kernel: [  81293]  1000 81293     2066      767        0      767         0    57344       96           200 su
Dec 23 12:13:33 gentoo kernel: [  81303]     0 81303     3342      874        1      873         0    69632     1472           200 bash
Dec 23 12:13:33 gentoo kernel: [  81419]     0 81419   214637     6294       64     6230         0   987136      192           200 journalctl
Dec 23 12:13:33 gentoo kernel: [  81965]  1000 81965   771265    39571    11373    28198         0  3452928   106239           100 Isolated Web Co
Dec 23 12:13:33 gentoo kernel: [  82206]  1000 82206   625590    22442      744    21698         0   774144     7743           100 Isolated Web Co
Dec 23 12:13:33 gentoo kernel: [  84742]  1000 84742   610500    11462       75    11371        16   479232     3296           233 Web Content
Dec 23 12:13:33 gentoo kernel: [  85142]  1000 85142   610500    11836      398    11417        21   487424     3040           233 Web Content
Dec 23 12:13:33 gentoo kernel: [  90419]     0 90419      646      376        0      376         0    45056       32             0 inotifywait
Dec 23 12:13:33 gentoo kernel: [  94172]     0 94172     4550     1535        0     1535         0    81920     1536           200 python3.12
Dec 23 12:13:33 gentoo kernel: [  94173]     0 94173     4551     1545        0     1545         0    81920     1536           200 python3.12
Dec 23 12:13:33 gentoo kernel: [  94176]   250 94176      604      374        0      374         0    45056       32           200 sandbox
Dec 23 12:13:33 gentoo kernel: [  94177]   250 94177     3210      875        0      875         0    69632     1216           200 bash
Dec 23 12:13:33 gentoo kernel: [  94193]   250 94193     3215      611        0      611         0    65536     1248           200 bash
Dec 23 12:13:33 gentoo kernel: [  94219]   250 94219    21554      895        2      893         0   225280    19936           200 ninja
Dec 23 12:13:33 gentoo kernel: [ 107345]   250 107345     3679      505        0      505         0    69632      480           200 x86_64-pc-linux
Dec 23 12:13:33 gentoo kernel: [ 107400]   250 107400     2266      564        0      564         0    57344      256           200 x86_64-pc-linux
Dec 23 12:13:33 gentoo kernel: [ 107401]   250 107401  1119335   933149   930026     3123         0  8974336   159155           200 cc1plus
Dec 23 12:13:33 gentoo kernel: [ 107402]   250 107402     2549      541        0      541         0    61440      192           200 as
Dec 23 12:13:33 gentoo kernel: [ 108109]  1000 108109     2878      867        0      867         0    61440      992           200 bash
Dec 23 12:13:33 gentoo kernel: [ 108500]  1000 108500   610500    13961       97    13864         0   483328     3264           233 Web Content
Dec 23 12:13:33 gentoo kernel: [ 108859]   250 108859     3514     2365        0     2365         0    77824      352           200 x86_64-pc-linux
Dec 23 12:13:33 gentoo kernel: [ 108877]   250 108877     2266      780       32      748         0    61440      256           200 x86_64-pc-linux
Dec 23 12:13:33 gentoo kernel: [ 108878]   250 108878   550147   382065   375082     6983         0  4345856   142272           200 cc1plus
Dec 23 12:13:33 gentoo kernel: [ 108879]   250 108879     3473     1854      960      894         0    73728      160           200 as
Dec 23 12:13:33 gentoo kernel: [ 108958]   250 108958     3514     2320        4     2316         0    69632      352           200 x86_64-pc-linux
Dec 23 12:13:33 gentoo kernel: [ 108973]   250 108973     2266      775       32      743         0    61440      256           200 x86_64-pc-linux
Dec 23 12:13:33 gentoo kernel: [ 108974]   250 108974   488514   399255   393285     5970         0  3862528    60896           200 cc1plus
Dec 23 12:13:33 gentoo kernel: [ 108976]   250 108976     2549      862        0      862         0    65536      192           200 as
Dec 23 12:13:33 gentoo kernel: [ 109030]   250 109030     3552     2417        2     2415         0    77824      352           200 x86_64-pc-linux
Dec 23 12:13:33 gentoo kernel: [ 109061]   250 109061     2266      721        0      721         0    65536      256           200 x86_64-pc-linux
Dec 23 12:13:33 gentoo kernel: [ 109062]   250 109062   361225   314781   310048     4733         0  2760704     6112           200 cc1plus
Dec 23 12:13:33 gentoo kernel: [ 109063]   250 109063     2549      936        0      936         0    57344      192           200 as
Dec 23 12:13:33 gentoo kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/user.slice/user-1000.slice/[email protected]/app.slice/[email protected],task=cc1plus,pid=107401,uid=250
Dec 23 12:13:33 gentoo kernel: Out of memory: Killed process 107401 (cc1plus) total-vm:4477340kB, anon-rss:3720104kB, file-rss:12492kB, shmem-rss:0kB, UID:250 pgtables:8764kB oom_score_adj:200
Dec 23 12:13:33 gentoo systemd[1]: [email protected]: A process of this unit has been killed by the OOM killer.

Config:

## This is the configuration file of the nohang daemon.

## The configuration includes the following sections:
##  0. Check kernel messages for OOM events
##  1. Common zram settings
##  2. Common PSI settings
##  3. Poll rate
##  4. Warnings and notifications
##  5. Soft (SIGTERM) threshold
##  6. Hard (SIGKILL) threshold
##  7. Customize victim selection: adjusting badness of processes
##  8. Customize soft corrective actions
##  9. Misc settings
## 10. Verbosity, debug, logging

## WARNING!
##  - Lines starting with #, tabs and whitespace characters are comments.
##  - Lines starting with @ contain optional parameters that may be repeated.
##  - All values are case sensitive.
##  - nohang doesn't forbid you to shoot yourself in the foot. Be careful!
##  - Restart the daemon after editing the file to apply the new settings.
##  - You can find the file with default values here: /usr/share/nohang/nohang.conf

## To find config keys descriptions see man(8) nohang

###############################################################################

##  0. Check kernel messages for OOM events

#   @check_kmsg
##  Type: boolean
##  Comment/uncomment to disable/enable checking kmsg for OOM events

#   @debug_kmsg
##  Type: boolean
##  Comment/uncomment to disable/enable debug checking kmsg

###############################################################################

    1. Common zram settings

    Key: zram_checking_enabled
    Description:
    Type: boolean
    Valid values: True | False
    Default value: False

zram_checking_enabled = True

###############################################################################

    2. Common PSI settings

    Key: psi_checking_enabled
    Description:
    Type: boolean
    Valid values: True | False
    Default value: True

psi_checking_enabled = True

    Key: psi_path
    Description:
    Type: string
    Valid values: any string
    Default value: /proc/pressure/memory

psi_path = /proc/pressure/memory

    Key: psi_metrics
    Description:
    Type: string
    Valid values: some_avg10, some_avg60, some_avg300,
                  full_avg10, full_avg60, full_avg300
    Default value: full_avg10

psi_metrics = full_avg10

    Key: psi_excess_duration
    Description:
    Type: float
    Valid values: >= 0
    Default value: 30

psi_excess_duration = 30

    Key: psi_post_action_delay
    Description:
    Type: float
    Valid values: >= 10
    Default value: 15

psi_post_action_delay = 15

###############################################################################

    3. Poll rate

    Key: fill_rate_mem
    Description:
    Type: float
    Valid values: >= 100
    Default value: 6000

fill_rate_mem  = 6000

    Key: fill_rate_swap
    Description:
    Type: float
    Valid values: >= 100
    Default value: 2000

fill_rate_swap = 2000

    Key: fill_rate_zram
    Description:
    Type: float
    Valid values: >= 100
    Default value: 4000

fill_rate_zram = 4000

    Key: max_sleep
    Description:
    Type: float
    Valid values: >= 0.01 and >= min_sleep
    Default value: 3

max_sleep = 3

    Key: min_sleep
    Description:
    Type: float
    Valid values: >= 0.01 and <= max_sleep
    Default value: 0.1

min_sleep = 0.1

###############################################################################

    4. Warnings and notifications

    4.1. GUI notifications after corrective actions

    Key: post_action_gui_notifications
    Description:
    Type: boolean
    Valid values: True | False
    Default value: True

post_action_gui_notifications = True

    Key: hide_corrective_action_type
    Description:
    Type: boolean
    Valid values: True | False
    Default value: False

hide_corrective_action_type = False

    4.2. Low memory warnings

    Key: low_memory_warnings_enabled
    Description:
    Type: boolean
    Valid values: True | False
    Default value: True

low_memory_warnings_enabled = True

    Key: warning_exe
    Description:
    Type: string
    Valid values: any string
    Default value: (empty string)

warning_exe =

    Key: warning_threshold_min_mem
    Description:
    Type: float (with % or M)
    Valid values: from the range [0; 100] %
    Default value: 20 %

warning_threshold_min_mem  = 20 %

    Key: warning_threshold_min_swap
    Description:
    Type: float (with % or M)
    Valid values: [0; 100] % or >= 0 M
    Default value: 20 %

warning_threshold_min_swap = 25 %

    Key: warning_threshold_max_zram
    Description:
    Type: float (with % or M)
    Valid values: from the range [0; 100] %
    Default value: 45 %

warning_threshold_max_zram = 45 %

    Key: warning_threshold_max_psi
    Description:
    Type: float
    Valid values: from the range [0; 100]
    Default value: 10

warning_threshold_max_psi  = 10

    Key: min_post_warning_delay
    Description:
    Type: float
    Valid values: >= 1
    Default value: 60

min_post_warning_delay = 60

    Key: env_cache_time
    Description:
    Type: float
    Valid values: >= 0
    Default value: 300

env_cache_time = 300

###############################################################################

    5. Soft threshold (thresholds for sending the SIGTERM signal or
                       implementing other soft corrective action)

    Key: soft_threshold_min_mem
    Description:
    Type: float (with % or M)
    Valid values: from the range [0; 50] %
    Default value: 5 %

soft_threshold_min_mem = 5 %

    Key: soft_threshold_min_swap
    Description:
    Type: float (with % or M)
    Valid values: [0; 100] % or >= 0 M
    Default value: 10 %

soft_threshold_min_swap = 10 %

    Key: soft_threshold_max_zram
    Description:
    Type: float (with % or M)
    Valid values: from the range [10; 90] %
    Default value: 55 %

soft_threshold_max_zram = 55 %

    Key: soft_threshold_max_psi
    Description:
    Type: float
    Valid values: from the range [5; 100]
    Default value: 40

soft_threshold_max_psi  = 40

###############################################################################

    6. Hard threshold (thresholds for sending the SIGKILL signal)

    Key: hard_threshold_min_mem
    Description:
    Type: float (with % or M)
    Valid values: from the range [0; 50] %
    Default value: 2 %

hard_threshold_min_mem = 2 %

    Key: hard_threshold_min_swap
    Description:
    Type: float (with % or M)
    Valid values: [0; 100] % or >= 0 M
    Default value: 4 %

hard_threshold_min_swap = 4 %

    Key: hard_threshold_max_zram
    Description:
    Type: float (with % or M)
    Valid values: from the range [10; 90] %
    Default value: 60 %

hard_threshold_max_zram = 60 %

    Key: hard_threshold_max_psi
    Description:
    Type: float
    Valid values: from the range [5; 100]
    Default value: 90

hard_threshold_max_psi = 90

###############################################################################

    7. Customize victim selection: adjusting badness of processes

    7.1. Ignore positive oom_score_adj

    Key: ignore_positive_oom_score_adj
    Description:
    Type: boolean
    Valid values: True | False
    Default value: False

ignore_positive_oom_score_adj = False

    7.2.1. Matching process names with RE patterns change their badness

    Syntax:

    @BADNESS_ADJ_RE_NAME  badness_adj  ///  RE_pattern

    New badness value will be += badness_adj

    It is possible to compare multiple patterns
    with different badness_adj values.

    Example:
    @BADNESS_ADJ_RE_NAME -500 /// ^sshd$

    Prefer terminating Firefox tabs instead of terminating the entire browser.
    (In Chromium and Electron-based apps child processes get oom_score_adj=300
    by default.)
@BADNESS_ADJ_RE_NAME   200  ///  ^(Web Content|Privileged Cont|file:// Content)$

@BADNESS_ADJ_RE_NAME  -200  ///  ^(dnf|yum|packagekitd)$


    7.2.2. Matching CGroup_v1-line with RE patterns

    @BADNESS_ADJ_RE_CGROUP_V1 -50 ///  ^/system\.slice/

    @BADNESS_ADJ_RE_CGROUP_V1  50 ///  /foo\.service$

    @BADNESS_ADJ_RE_CGROUP_V1 -50 ///  ^/user\.slice/

    7.2.3. Matching CGroup_v2-line with RE patterns

    @BADNESS_ADJ_RE_CGROUP_V2  100 /// ^/workload

    7.2.4. Matching eUIDs with RE patterns

    @BADNESS_ADJ_RE_UID -100 /// ^0$

    7.2.5. Matching /proc/[pid]/exe realpath with RE patterns

    Example:
    @BADNESS_ADJ_RE_REALPATH  20  ///  ^/usr/bin/foo$

    Protect X.
@BADNESS_ADJ_RE_REALPATH -200  ///  ^(/usr/libexec/Xorg|/usr/lib/xorg/Xorg|/usr/lib/Xorg|/usr/bin/X|/usr/bin/Xorg|/usr/bin/Xwayland|/usr/bin/weston|/usr/bin/sway)$

    Protect GNOME.
@BADNESS_ADJ_RE_REALPATH -200  ///  ^(/usr/bin/gnome-shell|/usr/bin/metacity|/usr/bin/mutter|/usr/lib/gnome-session/gnome-session-binary|/usr/libexec/gnome-session-binary|/usr/libexec/gnome-session-ctl)$

    Protect KDE Plasma.
@BADNESS_ADJ_RE_REALPATH -200  ///  ^(/usr/bin/plasma-desktop|/usr/bin/plasmashell|/usr/bin/plasma_session|/usr/bin/kwin|/usr/bin/kwin_x11|/usr/bin/kwin_wayland)$
@BADNESS_ADJ_RE_REALPATH -200  ///  ^(/usr/bin/startplasma-wayland|/usr/lib/x86_64-linux-gnu/libexec/startplasma-waylandsession|/usr/bin/ksmserver)$

    Protect Cinnamon.
@BADNESS_ADJ_RE_REALPATH -200  ///  ^(/usr/bin/cinnamon|/usr/bin/muffin|/usr/bin/cinnamon-session|/usr/bin/cinnamon-launcher)$

    Protect Xfce.
@BADNESS_ADJ_RE_REALPATH -200  ///  ^(/usr/bin/xfwm4|/usr/bin/xfce4-session|/usr/bin/xfce4-panel|/usr/bin/xfdesktop)$

    Protect Mate.
@BADNESS_ADJ_RE_REALPATH -200  ///  ^(/usr/bin/marco|/usr/bin/mate-session|/usr/bin/caja|/usr/bin/mate-panel)$

    Protect LXQt.
@BADNESS_ADJ_RE_REALPATH -200  ///  ^(/usr/bin/lxqt-panel|/usr/bin/pcmanfm-qt|/usr/bin/lxqt-session)$

    Protect Budgie Desktop.
@BADNESS_ADJ_RE_REALPATH -200  ///  ^(/usr/bin/budgie-wm|/usr/bin/budgie-panel)$

    Protect other.
@BADNESS_ADJ_RE_REALPATH -200  ///  ^(/usr/bin/compiz|/usr/bin/openbox|/usr/bin/fluxbox|/usr/bin/awesome|/usr/bin/icewm|/usr/bin/enlightenment|/usr/bin/gala|/usr/bin/wingpanel|/usr/bin/i3)$
@BADNESS_ADJ_RE_REALPATH -200  ///  ^(/usr/bin/teams|/usr/bin/carla|/opt/teams/teams)$

    Protect display managers.
@BADNESS_ADJ_RE_REALPATH -200  ///  ^(/usr/sbin/gdm|/usr/sbin/gdm3|/usr/sbin/sddm|/usr/bin/sddm|/usr/lib/x86_64-linux-gnu/sddm/sddm-helper|/usr/bin/slim|/usr/sbin/lightdm|/usr/libexec/gdm-session-worker|/usr/libexec/gdm-wayland-session|/usr/lib/gdm3/gdm-wayland-session|/usr/lib/gdm3/gdm-session-worker)$
@BADNESS_ADJ_RE_REALPATH -200  ///  ^/usr/lib/gdm3/

    Protect systemd-logind.
@BADNESS_ADJ_RE_REALPATH -200  ///  ^(/lib/systemd/systemd-logind|/usr/lib/systemd/systemd-logind)$

    Protect `systemd --user`.
@BADNESS_ADJ_RE_REALPATH -200  ///  ^(/lib/systemd/systemd|/usr/lib/systemd/systemd)$

    Protect dbus.
@BADNESS_ADJ_RE_REALPATH -200  ///  ^(/usr/bin/dbus-daemon|/usr/bin/dbus-run-session|/usr/bin/dbus-broker-launcher|/usr/bin/dbus-broker)$

    Protect package managers and distro installers.
@BADNESS_ADJ_RE_REALPATH -200  ///  ^(/usr/bin/calamares|/usr/bin/dpkg|/usr/bin/pacman|/usr/bin/yay|/usr/bin/pamac|/usr/bin/pamac-daemon|/usr/bin/pamac-manager|/usr/bin/emerge)$

    Prefer stress.
    @BADNESS_ADJ_RE_REALPATH  900  ///  ^(/usr/bin/stress|/usr/bin/stress-ng)$


    7.2.6. Matching /proc/[pid]/cwd realpath with RE patterns

    @BADNESS_ADJ_RE_CWD  200  ///  ^/home/

    7.2.7. Matching cmdlines with RE patterns
    WARNING: using this option can greatly slow down the search for a victim
    in conditions of heavily swapping.

    Prefer Chromium tabs and Electron-based apps
    @BADNESS_ADJ_RE_CMDLINE  200 /// --type=renderer

    Prefer Firefox tabs (Web Content and WebExtensions)
    @BADNESS_ADJ_RE_CMDLINE  100 /// -appomni

    @BADNESS_ADJ_RE_CMDLINE -200 /// ^/usr/lib/virtualbox

    7.2.8. Matching environ with RE patterns
    WARNING: using this option can greatly slow down the search for a victim
    in conditions of heavily swapping.

    @BADNESS_ADJ_RE_ENVIRON 100 /// USER=user


    Note that you can control badness also via systemd units via
    OOMScoreAdjust, see
    www.freedesktop.org/software/systemd/man/systemd.exec.html#OOMScoreAdjust=

###############################################################################

    8. Customize soft corrective actions

    Run the command instead of sending a signal with at soft corrective action
    if the victim's name or cgroup matches the regular expression.

    Syntax:
    KEY                         REGEXP      SEPARATOR     COMMAND

    @SOFT_ACTION_RE_NAME       ^foo$             /// kill -USR1  $PID
    @SOFT_ACTION_RE_CGROUP_V1  ^/system\.slice/  /// systemctl restart $SERVICE
    @SOFT_ACTION_RE_CGROUP_V2  /foo\.service$    /// systemctl restart $SERVICE

    $PID will be replaced by process PID.
    $NAME will be replaced by process name.
    $SERVICE will be replaced by .service if it exists (overwise it will be
                                                        relpaced by empty line)

###############################################################################

    9. Misc settings

    Key: max_soft_exit_time
    Description:
    Type: float
    Valid values: >= 0.1
    Default value: 10

max_soft_exit_time = 10

    Key: post_kill_exe
    Description:
    Type: string
    Valid values: any string
    Default value: (empty string)

post_kill_exe =

    Key: min_badness
    Description:
    Type: integer
    Valid values: >= 1
    Default value: 1

min_badness = 1

    Key: post_soft_action_delay
    Description:
    Type: float
    Valid values: >= 0.1
    Default value: 3

post_soft_action_delay = 3

    Key: post_zombie_delay
    Description:
    Type: float
    Valid values: >= 0
    Default value: 0.1

post_zombie_delay = 0.1

    Key: victim_cache_time
    Description:
    Type: float
    Valid values: >= 0
    Default value: 10

victim_cache_time = 10

    Key: exe_timeout
    Description:
    Type: float
    Valid values: >= 0.1
    Default value: 20

exe_timeout = 20

###############################################################################

   10. Verbosity, debug, logging

    Key: print_config_at_startup
    Description:
    Type: boolean
    Valid values: True | False
    Default value: False

print_config_at_startup = False

    Key: print_mem_check_results
    Description:
    Type: boolean
    Valid values: True | False
    Default value: False

print_mem_check_results = True

    Key: min_mem_report_interval
    Description:
    Type: float
    Valid values: >= 0
    Default value: 60

min_mem_report_interval = 60

    Key: print_proc_table
    Description:
    Type: boolean
    Valid values: True | False
    Default value: False

print_proc_table = True

    Key: extra_table_info
    Description:
    WARNING: using "cmdline" or "environ" keys can greatly slow down
    the search for a victim in conditions of heavy swapping.
    Type: string
    Valid values: None, cgroup_v1, cgroup_v2, realpath,
                  cwd, cmdline, environ
    Default value: None

extra_table_info = None

    Key: print_victim_status
    Description:
    Type: boolean
    Valid values: True | False
    Default value: True

print_victim_status = True

    Key: print_victim_cmdline
    Description:
    Type: boolean
    Valid values: True | False
    Default value: False

print_victim_cmdline = False

    Key: max_victim_ancestry_depth
    Description:
    Type: integer
    Valid values: >= 1
    Default value: 3

max_victim_ancestry_depth = 3

    Key: print_statistics
    Description:
    Type: boolean
    Valid values: True | False
    Default value: True

print_statistics = True

    Key: debug_psi
    Description:
    Type: boolean
    Valid values: True | False
    Default value: False

debug_psi = False

    Key: debug_gui_notifications
    Description:
    Type: boolean
    Valid values: True | False
    Default value: False

debug_gui_notifications = True

    Key: debug_sleep
    Description:
    Type: boolean
    Valid values: True | False
    Default value: False

debug_sleep = False

    Key: debug_threading
    Description:
    Type: boolean
    Valid values: True | False
    Default value: False

debug_threading = False

    Key: separate_log
    Description:
    Type: boolean
    Valid values: True | False
    Default value: False

separate_log = False

###############################################################################

    Use cases, feature requests and any questions are welcome:
    https://github.com/hakavlad/nohang/issues
@hakavlad
Copy link
Owner

nohang-desktop stopped working?

I can't say that nohang has stopped working. I can assume that kernel's OOM killer triggered prematurely. We need to find out why.

@hakavlad
Copy link
Owner

Dec 23 12:13:22 gentoo nohang-desktop[81157]: PSI: 7.95 | MemAvail: 9149 M, 39.0 % | SwapFree: 654 M, 11.1 %

then

Dec 23 12:13:33 gentoo kernel: kswapd0 invoked oom-killer:

@hakavlad
Copy link
Owner

How can I debug this further?

Maybe try

min_mem_report_interval = 0

to check (and log) memory more often.

@hakavlad
Copy link
Owner

Is MGLRU enabled?

With MGLRU enabled, the killer may be more aggressive.

@hakavlad
Copy link
Owner

hakavlad commented Dec 23, 2024 •
edited
Loading

I reproduced this with MGLRU: with nohang disabled, killer killed with a lot of SwapfFree even with min_ttl_ms=0.

@dim-geo
Copy link
Author

dim-geo commented Dec 23, 2024 •
edited
Loading

MGLRU was enabled. Disabled MGLRU with set_mglru 0 and run again nohang --memload. OOM killer was triggered again before nohang. (I already set min_mem_report_interval = 1)

@dim-geo
Copy link
Author

dim-geo commented Dec 23, 2024

Warnings are triggered but no action is taken.

Dec 23 16:45:35 gentoo nohang-desktop[25555]: Executing Command-63 ['sudo', '-u', 'none', 'env', 'DISPLAY=:0', 'DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus', 'notify-send', '--icon=dialog-warning', '--app-name=noha>
Dec 23 16:45:35 gentoo nohang-desktop[25555]: Command-63 execution completed in 0.068s; exit status: 0
Dec 23 16:45:35 gentoo nohang-desktop[25555]: PSI:  14.73 | MemAvail:  3395 M, 14.5 % | SwapFree:   507 M,   2.2 % | MemUsedZram:   429 M,  1.8 % | dMem:  -659 M/s
Dec 23 16:45:35 gentoo nohang-desktop[25555]: Warning threshold exceeded
Dec 23 16:45:35 gentoo nohang-desktop[25555]: Memory info, MiB:
Dec 23 16:45:35 gentoo nohang-desktop[25555]:   total=23489, used=19697, free=120, available=3395, shared=28, buffers=7, cache=3666,
Dec 23 16:45:35 gentoo nohang-desktop[25555]:   swap_total=23489, swap_used=22983, swap_free=506
Dec 23 16:45:35 gentoo nohang-desktop[25555]: Memory pressure (system-wide):
Dec 23 16:45:35 gentoo nohang-desktop[25555]:   some avg10=15.11 avg60=7.58 avg300=3.46
Dec 23 16:45:35 gentoo nohang-desktop[25555]:   full avg10=14.73 avg60=7.28 avg300=3.30
Dec 23 16:45:35 gentoo nohang-desktop[25555]: Found env in 0 ms (cached)
Dec 23 16:45:35 gentoo nohang-desktop[25555]:   Title:    ['Low memory']
Dec 23 16:45:35 gentoo nohang-desktop[25555]:   Body:     ['Save your unsaved data!\nClose unused apps!']
Dec 23 16:45:35 gentoo nohang-desktop[25555]:   Env list: [('none', 'DISPLAY=:0', 'DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus')]
Dec 23 16:45:35 gentoo sudo[28055]:     root : PWD=/ ; USER=none ; COMMAND=/usr/sbin/env DISPLAY=:0 DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus notify-send --icon=dialog-warning --app-name=nohang 'Low memory' 'Save>
Dec 23 16:45:35 gentoo sudo[28055]: pam_unix(sudo:session): session opened for user none(uid=1000) by (uid=0)
Dec 23 16:45:35 gentoo sudo[28055]: pam_unix(sudo:session): session closed for user none
Dec 23 16:45:35 gentoo kernel: python3 invoked oom-killer: gfp_mask=0x140dca(GFP_HIGHUSER_MOVABLE|__GFP_COMP|__GFP_ZERO), order=0, oom_score_adj=200
Dec 23 16:45:36 gentoo kernel: CPU: 3 UID: 1000 PID: 27562 Comm: python3 Tainted: G S      W  OE      6.12.6-cachyos-6.12.6-x86_64 #1
Dec 23 16:45:36 gentoo kernel: Tainted: [S]=CPU_OUT_OF_SPEC, [W]=WARN, [O]=OOT_MODULE, [E]=UNSIGNED_MODULE
Dec 23 16:45:36 gentoo kernel: Hardware name: Gigabyte Technology Co., Ltd. H97M-D3H/H97M-D3H, BIOS F7 08/03/2015
Dec 23 16:45:36 gentoo kernel: Sched_ext: flash (enabled+all), task: runnable_at=+0ms
Dec 23 16:45:36 gentoo kernel: Call Trace:
Dec 23 16:45:36 gentoo kernel:  <TASK>
Dec 23 16:45:36 gentoo kernel:  dump_stack_lvl+0x56/0x80
Dec 23 16:45:36 gentoo kernel:  dump_header+0x50/0x36d
Dec 23 16:45:36 gentoo kernel:  ? ___ratelimit+0x99/0xf0
Dec 23 16:45:36 gentoo kernel:  oom_kill_process.cold+0x8/0x8a
Dec 23 16:45:36 gentoo kernel:  out_of_memory+0x102/0x410
Dec 23 16:45:36 gentoo kernel:  __alloc_pages_slowpath.constprop.2+0x82f/0xce0
Dec 23 16:45:36 gentoo kernel:  __alloc_pages_noprof+0x2ec/0x300
Dec 23 16:45:36 gentoo kernel:  __folio_alloc_noprof+0x10/0x30
Dec 23 16:45:36 gentoo kernel:  do_anonymous_page+0x2ef/0x7b0
Dec 23 16:45:36 gentoo kernel:  ? __pte_offset_map+0x17/0x170
Dec 23 16:45:36 gentoo kernel:  __handle_mm_fault+0xa24/0xaa0
Dec 23 16:45:36 gentoo kernel:  handle_mm_fault+0xd9/0x2d0
Dec 23 16:45:36 gentoo kernel:  do_user_addr_fault+0x214/0x850
Dec 23 16:45:36 gentoo kernel:  exc_page_fault+0x6f/0x160
Dec 23 16:45:36 gentoo kernel:  asm_exc_page_fault+0x22/0x30
Dec 23 16:45:36 gentoo kernel: RIP: 0033:0x74bd9b2d1682
Dec 23 16:45:36 gentoo kernel: Code: 4c 89 ee 49 39 d0 49 89 48 60 0f 95 c2 48 29 d8 48 83 c7 10 0f b6 d2 48 83 c8 01 48 c1 e2 02 48 09 da 48 83 ca 01 48 89 57 f8 <48> 89 41 08 48 89 3c 24 e8 21 c7 ff ff 48 8b 3c 24 e9 a6 f7 >
Dec 23 16:45:36 gentoo kernel: RSP: 002b:00007fff3d848220 EFLAGS: 00010206
Dec 23 16:45:36 gentoo kernel: RAX: 000000000000c971 RBX: 000000000000a010 RCX: 000060a931eff690
Dec 23 16:45:36 gentoo kernel: RDX: 000000000000a011 RSI: 000000000000a001 RDI: 000060a931ef5690
Dec 23 16:45:36 gentoo kernel: RBP: ffffffffffffffb0 R08: 000074bd9b3f6ac0 R09: 0000000000000004
Dec 23 16:45:36 gentoo kernel: R10: 0000000001ffffff R11: 0000000000000002 R12: 0000000000000002
Dec 23 16:45:36 gentoo kernel: R13: 000000000000a001 R14: 0000000000000000 R15: 000074bd9b3f6b20
Dec 23 16:45:36 gentoo kernel:  </TASK>

@hakavlad
Copy link
Owner

You can log each memory check:

min_mem_report_interval = 0

You can set more aggressive settings, increase the frequency of memory checks:

fill_rate_mem = 8000
fill_rate_swap = 4000
min_sleep = 0.02

@hakavlad
Copy link
Owner

Question: is it nohang missing the state when it needs to react, or is it the kernel killing the process prematurely, before the state at which nohang starts to react.

@dim-geo
Copy link
Author

dim-geo commented Dec 23, 2024

I see that nohang understands that there is a problem ongoing, but it doesn't react fast enough. OOM kills it first.
Even with these settings.

fill_rate_mem = 100
fill_rate_swap = 100
min_sleep = 0.02

@hakavlad
Copy link
Owner

fill_rate_mem = 100
fill_rate_swap = 100

It's too low! It can react in time if the memory fills up at a speed of up to 100 MiB/s, but not more (but checks beetween min_sleep and max_sleep).

@dim-geo
Copy link
Author

dim-geo commented Dec 23, 2024

Increased the fill_rate, but no change :(

@hakavlad
Copy link
Owner

run again nohang --memload. OOM killer was triggered

This shouldn't have happened. This command does not cause an OOM, it terminates itself if no one interferes with it.

# nohang --memload
Enter the numbers 381268 to confirm that you are not a robot: 381268
--------------------------------------------------------------------
Warning! The process will consume memory until 40 MiB of memory
(MemAvailable + SwapFree) remain free, and it will be terminated via SIGUSR1
at the end. This may cause the system to freeze and processes to terminate.
Do you want to continue? [No/Yes] Yes
Memory consumption has started!
MemAvailable: 34 MiB, SwapFree: 5 MiB                
Self terminating by SIGUSR1
User defined signal 1

@hakavlad
Copy link
Owner

  • Try after swapoff
  • What is your fs? (ZFS uses Adaptive Replacement Cache)
  • What happens when other kernel versions used

@dim-geo
Copy link
Author

dim-geo commented Dec 24, 2024 via email

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@dim-geo @hakavlad