Very high CPU usage from Wowza start

Wowza Streaming Engine
1

Recently my Wowza server has been using significantly higher amounts of CPU than before and is effectively unresponsive. Restarting doesn’t help, as CPU usage climbs within seconds of restart back to its high level.

From top:

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND

20863 root 20 0 3379m 65m 9940 S 369 1.7 9:27.63 java

Is there any reason why this may have popped up recently? Could it be due to problems with the streams? I’ve got 7 low quality streams coming in and am doing audio-only transcoding… This is on a Xeon E31270 @ 3.40GHz with 4 GB of RAM, and which previously seemed sufficient for the task…

Any thoughts? Thanks.

2

Reboot! (the whole computer)

3

lynxorg,

Based on the data of pr1001’s post, this problem was likely related to the July 1st leap second Java bug. If you are seeing Java problems after this date, please reboot. Restarting Wowza (the Java process) might also fix it, but I read that rebooting may be better.

4

Hey pr1001 ,

can you print here the whole top output ?

5

I don’t know whether to be happy or sad to be able to report that rebooting the machine seems to have done the trick. ;-p

6

Sorry, I don’t have it anymore.

7

I had the same issue. reboot fixed it, but http://blog.wpkg.org/2012/07/01/java-leap-second-bug-30-june-1-july-2012-fix/

suggests that you can simply check if you have something like the following in your dmesg:

[10703552.860274] Clock: inserting leap second 23:59:60 UTC

If you do, then this may work without forcing you to restart your server:

/etc/init.d/ntp stop
date -s "$(date)"

(the other suggested way around the web would be

date `date +"%m%d%H%M%C%y.%S"`

, having the same effect).

lynxorg: here was my top output with the H flag

EDIT (I wasn’t clear on this … but this output was after having restarted wowza, 4 cores and ~4G dedicated to java)

top - 16:06:01 up 101 days,  7:15,  2 users,  load average: 16.99, 16.91, 16.15
Tasks: 612 total,  12 running, 600 sleeping,   0 stopped,   0 zombie
Cpu(s):  7.5%us,  3.4%sy,  0.1%ni, 88.8%id,  0.0%wa,  0.0%hi,  0.2%si,  0.0%st
Mem:  16470024k total,  8970888k used,  7499136k free,   382104k buffers
Swap: 36130808k total,     3556k used, 36127252k free,  5907380k cached
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                                                           
11354 root      20   0 4807m 119m 9964 R   71  0.7   6:31.70 java                                                                                                                                              
11350 root      20   0 4807m 119m 9964 S   70  0.7   6:33.22 java                                                                                                                                              
11352 root      20   0 4807m 119m 9964 R   70  0.7   7:08.86 java                                                                                                                                              
11358 root      20   0 4807m 119m 9964 S   47  0.7   6:45.49 java                                                                                                                                              
11355 root      20   0 4807m 119m 9964 R   43  0.7   6:29.44 java                                                                                                                                              
11351 root      20   0 4807m 119m 9964 R   36  0.7   6:14.24 java                                                                                                                                              
11356 root      20   0 4807m 119m 9964 R   36  0.7   6:57.62 java                                                                                                                                              
11366 root      20   0 4807m 119m 9964 R   36  0.7   6:37.65 java                                                                                                                                              
11353 root      20   0 4807m 119m 9964 R   34  0.7   6:50.45 java                                                                                                                                              
11805 www-data  20   0  509m 310m 4504 S   34  1.9   0:32.86 apache2                                                                                                                                           
    4 root      20   0     0    0    0 R   30  0.0   1955:45 ksoftirqd/0                                                                                                                                       
   10 root      20   0     0    0    0 S   30  0.0   1879:27 ksoftirqd/2                                                                                                                                       
11339 root      20   0 4807m 119m 9964 S   30  0.7   4:50.46 java                                                                                                                                              
    7 root      20   0     0    0    0 S   28  0.0   1966:25 ksoftirqd/1                                                                                                                                       
   19 root      20   0     0    0    0 S   28  0.0   1948:41 ksoftirqd/5                                                                                                                                       
   25 root      20   0     0    0    0 S   26  0.0   1941:57 ksoftirqd/7                                                                                                                                       
11347 root      20   0 4807m 119m 9964 R   26  0.7   6:30.97 java                                                                                                                                              
11357 root      20   0 4807m 119m 9964 R   26  0.7   6:24.49 java                                                                                                                                              
11666 www-data  20   0  316m 119m 4516 R   24  0.7   0:08.09 apache2                                                                                                                                           
   22 root      20   0     0    0    0 S   23  0.0   1939:46 ksoftirqd/6                                                                                                                                       
   13 root      20   0     0    0    0 S   21  0.0   1511:35 ksoftirqd/3                                                                                                                                       
   16 root      20   0     0    0    0 S   21  0.0   1942:46 ksoftirqd/4