Java/Wowza high cpu load (1200%)

Wowza Streaming Engine
1

Hello,

this is a system running wowza. At random times (not always under high traffic) it goes fully overloaded due java/wowza.

Only way to fix it is restart wowza.

Versions:

  • Wowza Media Server 2.2.4 build27452 (25 April 2011)

  • java version “1.6.0_25” - Java™ SE Runtime Environment (build 1.6.0_25-b06) - Java HotSpot™ 64-Bit Server VM (build 20.0-b11, mixed mode)

Streaming:

  • live streaming (webcams)

top - 11:53:52 up 5 days, 1:37, 1 user, load average: 11.35, 12.00, 12.29

Tasks: 205 total, 2 running, 203 sleeping, 0 stopped, 0 zombie

Cpu0 : 0.0%us,100.0%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st

Cpu1 :100.0%us, 0.0%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st

Cpu2 :100.0%us, 0.0%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st

Cpu3 :100.0%us, 0.0%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st

Cpu4 :100.0%us, 0.0%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st

Cpu5 :100.0%us, 0.0%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st

Cpu6 :100.0%us, 0.0%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st

Cpu7 :100.0%us, 0.0%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st

Cpu8 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st

Cpu9 :100.0%us, 0.0%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st

Cpu10 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st

Cpu11 :100.0%us, 0.0%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st

Cpu12 :100.0%us, 0.0%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st

Cpu13 :100.0%us, 0.0%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st

Cpu14 :100.0%us, 0.0%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st

Cpu15 :100.0%us, 0.0%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st

Mem: 24674040k total, 19957468k used, 4716572k free, 455124k buffers

Swap: 6144856k total, 0k used, 6144856k free, 14172028k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND

17083 root 20 0 7513m 1.4g 10m S 1281.2 5.9 9251:06 java

This is now with 6 or 7 live streams and about 50 viewers. (Only restarting wowza would set load to 0.xx again)

2

I would post your Application.xml and possibly your Server.xml along with finding a point in your logs where the issue starts, may show an error/output.

Shamrock

3

A Wowza production server has to be tuned:

https://www.wowza.com/docs/how-to-do-performance-tuning

Monitor with JConsole, because Wowza/Java reserves ram with java heap size (see tuning guide).

https://www.wowza.com/docs/quick-start-guides

Richard

4

Can you show the startup lines so we can see resources and tuning.

You might have your Java heap size set too high, not allowing enough for your system.

Richard

5

Paste in the most recent startup lines from your log file here, it shows environment and tuning.

Have you followed the tuning guide?

https://www.wowza.com/docs/how-to-do-performance-tuning

It usually turns out not to be the same thing, so can you describe your problem?

Richard

6

Great, very glad it’s working, and thanks for the update: very useful bit of info.

Richard

7

Access and Error Logs are clean of errors.

System logs are also clean.

Application.xml

<Root>
        <Application>
                <!-- Uncomment to set application level timeout values
                <ApplicationTimeout>60000</ApplicationTimeout>
                <PingTimeout>12000</PingTimeout>
                <ValidationFrequency>8000</ValidationFrequency>
                <MaximumPendingWriteBytes>0</MaximumPendingWriteBytes>
                <MaximumSetBufferTime>60000</MaximumSetBufferTime>
                <MaximumStorageDirDepth>25</MaximumStorageDirDepth>
                -->
                <Connections>
                        <AutoAccept>true</AutoAccept>
                        <AllowDomains></AllowDomains>
                </Connections>
                <!--
                        StorageDir path variables
                        ${com.wowza.wms.AppHome} - Application home directory
                        ${com.wowza.wms.ConfigHome} - Configuration home directory
                        ${com.wowza.wms.context.VHost} - Virtual host name
                        ${com.wowza.wms.context.VHostConfigHome} - Virtual host config directory
                        ${com.wowza.wms.context.Application} - Application name
                        ${com.wowza.wms.context.ApplicationInstance} - Application instance name
                -->
                <Streams>
                        <StreamType>default</StreamType>
                        <StorageDir>${com.wowza.wms.context.VHostConfigHome}/content</StorageDir>
                        <KeyDir>${com.wowza.wms.context.VHostConfigHome}/keys</KeyDir>
                        <!-- LiveStreamPacketizers (separate with commas): cupertinostreamingpacketizer, smoothstreamingpacketizer, sanjosestreamingpacketizer, cupertinostreamingrepeater, smoothstreamingrepeater, sanjosestreamingrepeater -->
                        <LiveStreamPacketizers></LiveStreamPacketizers>
                        <!-- Properties defined here will override any properties defined in conf/Streams.xml for any streams types loaded by this application -->
                        <Properties>
                        </Properties>
                </Streams>
                <!-- HTTPStreamers (separate with commas): cupertinostreaming, smoothstreaming, sanjosestreaming -->
                <HTTPStreamers>cupertinostreaming,smoothstreaming,sanjosestreaming</HTTPStreamers>
                <SharedObjects>
                        <StorageDir></StorageDir>
                </SharedObjects>
                <Client>
                        <IdleFrequency>-1</IdleFrequency>
                        <Access>
                                <StreamReadAccess>*</StreamReadAccess>
                                <StreamWriteAccess>*</StreamWriteAccess>
                                <StreamAudioSampleAccess></StreamAudioSampleAccess>
                                <StreamVideoSampleAccess></StreamVideoSampleAccess>
                                <SharedObjectReadAccess>*</SharedObjectReadAccess>
                                <SharedObjectWriteAccess>*</SharedObjectWriteAccess>
                        </Access>
                </Client>
                <RTP>
                        <!-- RTP/Authentication/[type]Methods defined in Authentication.xml. Default setup includes; none, basic, digest -->
                        <Authentication>
                                <PublishMethod>digest</PublishMethod>
                                <PlayMethod>digest</PlayMethod>
                        </Authentication>
                        <!-- RTP/AVSyncMethod. Valid values are: senderreport, systemclock, rtptimecode -->
                        <AVSyncMethod>senderreport</AVSyncMethod>
                        <MaxRTCPWaitTime>12000</MaxRTCPWaitTime>
                        <IdleFrequency>75</IdleFrequency>
                        <RTSPSessionTimeout>90000</RTSPSessionTimeout>
                        <RTSPMaximumPendingWriteBytes>0</RTSPMaximumPendingWriteBytes>
                        <RTSPBindIpAddress></RTSPBindIpAddress>
                        <RTSPConnectionIpAddress>0.0.0.0</RTSPConnectionIpAddress>
                        <RTSPOriginIpAddress>127.0.0.1</RTSPOriginIpAddress>
                        <IncomingDatagramPortRanges>*</IncomingDatagramPortRanges>
                        <!-- Properties defined here will override any properties defined in conf/RTP.xml for any depacketizers loaded by this application -->
                        <Properties>
                        </Properties>
                </RTP>
                <MediaCaster>
                        <!-- Properties defined here will override any properties defined in conf/MediaCasters.xml for any MediaCasters loaded by this applications -->
                        <Properties>
                        </Properties>
                </MediaCaster>
                <MediaReader>
                        <!-- Properties defined here will override any properties defined in conf/MediaReaders.xml for any MediaReaders loaded by this applications -->
                        <Properties>
                        </Properties>
                </MediaReader>
                <MediaWriter>
                        <!-- Properties defined here will override any properties defined in conf/MediaWriter.xml for any MediaWriter loaded by this applications -->
                        <Properties>
                        </Properties>
                </MediaWriter>
                <LiveStreamPacketizer>
                        <!-- Properties defined here will override any properties defined in conf/LiveStreamPacketizers.xml for any LiveStreamPacketizers loaded by this applications -->
                        <Properties>
                        </Properties>
                </LiveStreamPacketizer>
                <HTTPStreamer>
                        <!-- Properties defined here will override any properties defined in conf/HTTPStreamers.xml for any HTTPStreamer loaded by this applications -->
                        <Properties>
                        </Properties>
                </HTTPStreamer>
                <Repeater>
                        <OriginURL></OriginURL>
                        <QueryString><![CDATA[]]></QueryString>
                </Repeater> 
                <Modules>
                        <Module>
                                <Name>base</Name>
                                <Description>Base</Description>
                                <Class>com.wowza.wms.module.ModuleCore</Class>
                        </Module>
                        <Module>
                                <Name>properties</Name>
                                <Description>Properties</Description>
                                <Class>com.wowza.wms.module.ModuleProperties</Class>
                        </Module>
                        <Module>
                                <Name>logging</Name>
                                <Description>Client Logging</Description>
                                <Class>com.wowza.wms.module.ModuleClientLogging</Class>
                        </Module>
                        <Module>
                                <Name>flvplayback</Name>
                                <Description>FLVPlayback</Description>
                                <Class>com.wowza.wms.module.ModuleFLVPlayback</Class>
                        </Module> 
                </Modules>
                <!-- Properties defined here will be added to the IApplication.getProperties() and IApplicationInstance.getProperties() collections -->
                <Properties>
                </Properties>
        </Application>
</Root>

Server.xml

<Root>
        <Server>
                <CommandInterface>
                        <HostPort>
                                <IpAddress>127.0.0.1</IpAddress>
                                <Port>8083</Port>
                        </HostPort>
                </CommandInterface>
                <AdminInterface>
                        <!-- Objects exposed through JMX interface: Server, VHost, VHostItem, Application, ApplicationInstance, MediaCaster, Module, Client, MediaStream, SharedObject, Acceptor, IdleWorker -->
                        <ObjectList>Server,VHost,VHostItem,Application,ApplicationInstance,MediaCaster,Module,IdleWorker</ObjectList>
                </AdminInterface>
                <!-- JMXUrl: service:jmx:rmi://localhost:8084/jndi/rmi://localhost:8085/jmxrmi -->
                <JMXRemoteConfiguration>
                        <Enable>false</Enable>
                        <IpAddress>localhost</IpAddress> <!-- set to localhost or internal ip address if behind NAT -->
                        <RMIServerHostName>xxx.xxx.xxx.xxx</RMIServerHostName> <!-- set to external ip address or domain name if behind NAT -->
                        <RMIConnectionPort>8084</RMIConnectionPort>
                        <RMIRegistryPort>8085</RMIRegistryPort>
                        <Authenticate>true</Authenticate>
                        <PasswordFile>${com.wowza.wms.ConfigHome}/conf/jmxremote.password</PasswordFile>
                        <AccessFile>${com.wowza.wms.ConfigHome}/conf/jmxremote.access</AccessFile>
                        <SSLSecure>false</SSLSecure>
                </JMXRemoteConfiguration>
                <UserAgents>Shockwave Flash|CFNetwork|MacNetwork/1.0 (Macintosh)</UserAgents>
                <ServerListeners>
                        <!--
                        <ServerListener>
                                <BaseClass>com.wowza.wms.plugin.loadbalancer.ServerListenerLoadBalancerListener</BaseClass>
                        </ServerListener>
                        -->
                        <!--
                        <ServerListener>
                                <BaseClass>com.wowza.wms.plugin.loadbalancer.ServerListenerLoadBalancerSender</BaseClass>
                        </ServerListener>
                        -->
                </ServerListeners>
                <VHostListeners>
                        <!--
                        <VHostListener>
                                <BaseClass></BaseClass>
                        </VHostListener>
                        -->
                        <VHostListener>
                                <BaseClass>xxx.xxx.wms.common.listeners.ApplicationStartupListener</BaseClass>
                        </VHostListener>
                </VHostListeners> 
                <HandlerThreadPool>
                        <PoolSize>10</PoolSize>
                </HandlerThreadPool>
                <TransportThreadPool>
                        <PoolSize>10</PoolSize>
                </TransportThreadPool>
                <RTP>
                        <DatagramStartingPort>6970</DatagramStartingPort>
                        <DatagramPortSharing>false</DatagramPortSharing>
                </RTP>
                <!-- Properties defined here will be added to the IServer.getProperties() collection -->
                <Properties>
                </Properties>
        </Server>
</Root>
8

All settings are tunned but java goes to 1200% and load +10.00 with only ~1500 viewers.

Ram is enough (24gb total, 6gb free now with 10.00 load)

Will check with jconsole.

9

I haven’t found any solution…

This is a wowza dedicated server with enough cpu power and ram, tuned with performance tuning settings, only running wowza, but after some hours of use always goes overload:

top - 17:12:39 up 1 day, 18:19, 1 user, load average: 23.85, 24.98, 25.00

Tasks: 152 total, 1 running, 151 sleeping, 0 stopped, 0 zombie

Cpu0 : 82.2%us, 4.4%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 13.3%si, 0.0%st

Cpu1 :100.0%us, 0.0%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st

Cpu2 :100.0%us, 0.0%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st

Cpu3 : 97.7%us, 2.3%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st

Cpu4 : 88.9%us, 11.1%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st

Cpu5 : 93.3%us, 6.7%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st

Cpu6 : 95.7%us, 4.3%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st

Cpu7 :100.0%us, 0.0%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st

Mem: 24728664k total, 7130528k used, 17598136k free, 210056k buffers

Swap: 525308k total, 0k used, 525308k free, 4603396k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND

19979 root 20 0 13.2g 1.2g 9.9m S 796.9 5.0 2928:27 java

3531 root 20 0 12804 1056 740 R 2.2 0.0 0:01.10 top

1 root 20 0 10412 696 576 S 0.0 0.0 0:02.69 init

2 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kthreadd

3 root 20 0 0 0 0 S 0.0 0.0 0:01.46 ksoftirqd/0

10

I haven’t found any solution…

This is a wowza dedicated server with enough cpu power and ram, tuned with performance tuning settings, only running wowza, but after some hours of use always goes overload:

Any update on this? I’ve got a similar problem, with wowza v3, dedicated server with plenty of ram, and same behavior.

Was the problem the java heap size, as rrlanham suggested?

11

Paste in the most recent startup lines from your log file here, it shows environment and tuning.

Have you followed the tuning guide?

Tune Wowza Streaming Engine for performance

It usually turns out not to be the same thing, so can you describe your problem?

Richard

In our case, lowering down the heap size helped with the situation. We were having a high load with no apparent reason, and then the load would never go down.

We followed the tuning guide, and we had 8000M heap size on our 24G RAM dedicated machine. I lowered down the heap size to 6000M and everything seems fine.