rhel5下oom-killer

qingheliu

rhel下出现oom-killer
机器的内存及版本信息：
[root@I**** log]# free -m
             total       used       free     shared    buffers     cached
Mem:          4051       3815        236          0         27        320
-/+ buffers/cache:       3467        584
Swap:         8191          0       8191
[root@I**** log]# uname  -a
Linux I**** 2.6.18-128.el5PAE #1 SMP Wed Dec 17 12:02:33 EST 2008 i686 i686 i386 GNU/Linux
[root@I**** log]# cat /etc/issue
Red Hat Enterprise Linux Server release 5.3 (Tikanga)
Kernel \r on an \m

[root@I**** log]# cat /proc/meminfo  | grep -i lowfree
LowFree:        189044 kB
[root@I**** log]# cat  /proc/buddyinfo
Node 0, zone      DMA      3      3      4      4      3      2      1      0      1      1      2
Node 0, zone   Normal   1763   3337   2728    819    264    107     39     16      5      1      3
Node 0, zone  HighMem      1     79    191     20      0      1      1      0      0      0      0
[root@I**** log]#
机器中日志如下：

Aug 14 04:47:20 **** : error getting update info: Cannot retrieve repository metadata (repomd.xml) for repository: rhel5-Serve
r. Please verify its path and try again
Aug 14 05:55:58 **** kernel: jsvc invoked oom-killer: gfp_mask=0xd0, order=0, oomkilladj=0
Aug 14 05:58:10 **** kernel:  [<c0457a41>] out_of_memory+0x72/0x1a5
Aug 14 05:58:10 **** kernel:  [<c0458f16>] __alloc_pages+0x216/0x297
Aug 14 05:58:10 **** kernel:  [<c046da90>] cache_alloc_refill+0x26d/0x450
Aug 14 05:58:10 **** kernel:  [<c046d819>] kmem_cache_alloc+0x41/0x4b
Aug 14 05:58:10 **** kernel:  [<c047e030>] getname+0x1a/0xb0
Aug 14 05:58:10 **** kernel:  [<c046fdad>] do_sys_open+0x12/0xae
Aug 14 05:58:10 **** kernel:  [<c046fe76>] sys_open+0x16/0x18
Aug 14 05:58:10 **** kernel:  [<c0404f17>] syscall_call+0x7/0xb
Aug 14 05:58:10 **** kernel:  =======================
Aug 14 05:58:10 **** kernel: Mem-info:
Aug 14 05:58:10 **** kernel: DMA per-cpu:
Aug 14 06:17:44 **** : error getting update info: Cannot retrieve repository metadata (repomd.xml) for repository: rhel5-Serve
r. Please verify its path and try again
Aug 14 05:58:10 **** kernel: cpu 0 hot: high 0, batch 1 used:0
Aug 14 06:22:51 **** kernel: cpu 0 cold: high 0, batch 1 used:0
Aug 14 06:22:51 **** kernel: cpu 1 hot: high 0, batch 1 used:0
Aug 14 06:22:51 **** kernel: cpu 1 cold: high 0, batch 1 used:0
Aug 14 06:22:51 **** kernel: cpu 2 hot: high 0, batch 1 used:0
Aug 14 06:22:51 **** kernel: cpu 2 cold: high 0, batch 1 used:0
Aug 14 06:22:51 **** kernel: cpu 3 hot: high 0, batch 1 used:0
Aug 14 06:22:51 **** kernel: cpu 3 cold: high 0, batch 1 used:0
Aug 14 06:22:51 **** kernel: cpu 4 hot: high 0, batch 1 used:0
Aug 14 06:22:51 **** kernel: cpu 4 cold: high 0, batch 1 used:0
Aug 14 06:22:51 **** kernel: cpu 5 hot: high 0, batch 1 used:0
Aug 14 06:22:51 **** kernel: cpu 5 cold: high 0, batch 1 used:0
Aug 14 06:22:51 **** kernel: cpu 6 hot: high 0, batch 1 used:0
Aug 14 06:22:51 **** kernel: cpu 6 cold: high 0, batch 1 used:0
Aug 14 06:22:51 **** kernel: cpu 7 hot: high 0, batch 1 used:0
Aug 14 06:22:51 **** kernel: cpu 7 cold: high 0, batch 1 used:0
Aug 14 06:22:51 **** kernel: DMA32 per-cpu: empty
Aug 14 06:22:51 **** kernel: Normal per-cpu:
Aug 14 06:22:51 **** kernel: cpu 0 hot: high 186, batch 31 used:24
Aug 14 06:22:51 **** kernel: cpu 0 cold: high 62, batch 15 used:60
Aug 14 06:22:51 **** kernel: cpu 1 hot: high 186, batch 31 used:128
Aug 14 06:22:51 **** kernel: cpu 1 cold: high 62, batch 15 used:56
Aug 14 06:22:51 **** kernel: cpu 2 hot: high 186, batch 31 used:57
Aug 14 06:22:52 **** kernel: cpu 2 cold: high 62, batch 15 used:48
Aug 14 06:22:52 **** kernel: cpu 3 hot: high 186, batch 31 used:28
Aug 14 06:22:53 **** kernel: cpu 3 cold: high 62, batch 15 used:49
Aug 14 06:22:53 **** kernel: cpu 4 hot: high 186, batch 31 used:1
Aug 14 06:22:53 **** kernel: cpu 4 cold: high 62, batch 15 used:51
Aug 14 06:22:53 **** kernel: cpu 5 hot: high 186, batch 31 used:54

ug 14 06:22:56 **** kernel: Free pages:       43644kB (37380kB HighMem)
Aug 14 06:22:56 **** kernel: Active:534216 inactive:372949 dirty:209 writeback:279293 unstable:0 free:10911 slab:78781 mapped-
file:3189 mapped-anon:543544 pagetables:29744
Aug 14 06:22:56 **** kernel: DMA free:3572kB min:68kB low:84kB high:100kB active:48kB inactive:0kB present:16384kB pages_scann
ed:7992 all_unreclaimable? yes
Aug 14 06:22:56 **** kernel: lowmem_reserve[]: 0 0 880 4848
Aug 14 06:22:56 **** kernel: DMA32 free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_u
nreclaimable? no
Aug 14 06:22:56 **** kernel: lowmem_reserve[]: 0 0 880 4848
Aug 14 06:22:56 **** kernel: Normal free:2692kB min:3756kB low:4692kB high:5632kB active:48kB inactive:501308kB present:901120
kB pages_scanned:14069618 all_unreclaimable? yes
Aug 14 06:22:56 **** kernel: lowmem_reserve[]: 0 0 0 31744
Aug 14 06:22:56 **** kernel: HighMem free:37380kB min:512kB low:4748kB high:8988kB active:2136768kB inactive:990488kB present:
4063232kB pages_scanned:0 all_unreclaimable? no
Aug 14 06:22:56 **** kernel: lowmem_reserve[]: 0 0 0 0
Aug 14 06:22:56 **** kernel: DMA: 1*4kB 0*8kB 1*16kB 1*32kB 1*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 0*4096kB = 3572kB
Aug 14 06:22:56 **** kernel: DMA32: empty
Aug 14 06:22:56 **** kernel: Normal: 1*4kB 0*8kB 0*16kB 0*32kB 0*64kB 1*128kB 0*256kB 1*512kB 0*1024kB 1*2048kB 0*4096kB = 269
2kB
Aug 14 06:22:56 **** kernel: HighMem: 8637*4kB 300*8kB 3*16kB 0*32kB 0*64kB 1*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB
= 37380kB
Aug 14 06:22:56 **** kernel: 504858 pagecache pages
Aug 14 06:22:56 **** kernel: Swap cache: add 4824987, delete 4323819, find 8002987/8654496, race 45+525
Aug 14 06:22:56 **** kernel: Free swap  = 2510028kB
Aug 14 06:22:56 **** kernel: Total swap = 8388600kB
Aug 14 06:22:56 **** kernel: Free swap:       2510028kB
Aug 14 06:22:56 **** kernel: 1245184 pages of RAM
Aug 14 06:22:56 **** kernel: 1015808 pages of HIGHMEM
Aug 14 06:22:56 **** kernel: 207909 reserved pages
Aug 14 06:22:56 **** kernel: 1352825 pages shared
Aug 14 06:22:56 **** kernel: 501170 pages swap cached
Aug 14 06:22:56 **** kernel: 209 pages dirty
Aug 14 06:22:56 **** kernel: 279147 pages writeback  阿汽
Aug 14 06:22:56 **** kernel: 3189 pages mapped
Aug 14 06:22:56 **** kernel: 78781 pages slab
Aug 14 06:22:56 **** kernel: 29744 pages pagetables
Aug 14 06:22:56 **** kernel: Out of memory: Killed process 11747 (jsvc).
Aug 14 06:22:56 **** kernel: jsvc invoked oom-killer: gfp_mask=0xd0, order=0, oomkilladj=0
Aug 14 06:22:56 **** kernel:  [<c0457a41>] out_of_memory+0x72/0x1a5
Aug 14 06:22:56 **** kernel:  [<c0458f16>] __alloc_pages+0x216/0x297
Aug 14 06:22:56 **** kernel:  [<c046da90>] cache_alloc_refill+0x26d/0x450
Aug 14 06:22:56 **** kernel:  [<c046d819>] kmem_cache_alloc+0x41/0x4b
Aug 14 06:22:56 **** kernel:  [<c047e030>] getname+0x1a/0xb0
Aug 14 06:22:56 **** kernel:  [<c046fdad>] do_sys_open+0x12/0xae