Version: Wowza Media Server 2 Perpetual 2.2.4.07 build27628
OS: CentOS release 5.7 (Final) 64bit
RAM: 8GB
Java version: 1.6.0_20
Java options (WowzaMediaSever’s setenv): -Xmx5000M -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:+UseParNewGC -XX:NewSize=256m -Djava.net.preferIPv4Stack=true
The media server too much use memories. ( CPU total usage is usually under 12%)
This media server used only live streaming and VOD server is seperated.
I set a FMLE(windows 7 64bit) PC and FMLE sends stream(RTMP, H.264, AAC) to WowzaMediaServer.
At first, there was no problem. But there is freezing 1 or 2 days after.(memory usage is over 75%, the concurrent users are under 50. )
Then, decoder clients can’t play any streams.(RTSP, RTMP, HLS)
That’s why I restart WowzaMediaServer almost every 2 days.
I tested many times to solve this problem as performance tuning guide. (https://www.wowza.com/docs/how-to-do-performance-tuning)
The test has changed options.
But there still has same problem.
Is there the solution or hints?
If I upgrade WowzaMediaSever from 2 to 3, these problem will not appear?
Do you have any custom modules?
Are there any other applications on that server using substantial memory?
Show the start up lines in the access log.
Upgrading could help
https://www.wowza.com/docs/how-to-upgrade-wowza-media-server-software-to-wowza-streaming-engine
Richard
Try reducing java heap size to 4000M or 3500M, because you only have about 5g available when the server is started.
Hardware Physical Memory: 5311MB/7000MB
d
Richard
You are using advanced GC options which might not be helping. As stated in step #5 of the tuning guide, we suggest not using these options unless there is already a problem as they might cause problems. Do you have the “-server” flag turned on following step #4?
Have you followed tuning step #6 for a 12 core machine, which is what you have?
It is also possible that you are just over-loading the machine, in which case you might need to setup Liverepeater (origin/edge) cluster to handle the load.
Richard
Do you mean the /conf/[app-name]/Application.xml /RTP /Property “streamTimeout”?
<Property>
<Name>streamTimeout</Name>
<Value>12000</Value>
<Type>Integer</Type>
</Property>
Are there outOfMemory errors in the access and error log?
Is it possible for you to upgrade to Wowza 3.5? The current version is 3.5.0.02, with the latest patch applied.
https://www.wowza.com/docs/how-to-upgrade-wowza-media-server-software-to-wowza-streaming-engine
It might improve the issue at least, there has been much improvement since 2.2.4, though I cannot guarantee that
Richard
Richard, thank you for answer.
This machine dedicated for WowzaMediaServer. (Not VOD, but olny live with FMLE)
And I use only 1 module HTTPProvider, there is no other custom module.
Here is Start up lines.
2012-12-25 09:01:02 KST server-start server INFO 200 Wowza Media Server 2 Perpetual 2.2.4.07 build27628 - - - - 0.711 - - - - - - - - - - - - - - - - - - - - - - - - -
2012-12-25 09:01:02 KST comment server INFO 200 - Serial number: XXXXX-XXXXX-XXXXX-XXXXX-UCNRH - - - 0.713 - - - - - - - - - - - - - - - - - - - - - - - - -
2012-12-25 09:01:02 KST comment server INFO 200 - Maximum connections: Unlimited - - - 0.714 - - - - - - - - - - - - - - - - - - - - - - - - -
2012-12-25 09:01:02 KST comment server INFO 200 - Hardware Available Processors: 12 - - - 0.719 - - - - - - - - - - - - - - - - - - - - - - - - -
2012-12-25 09:01:02 KST comment server INFO 200 - Hardware Physical Memory: 5311MB/7000MB - - - 0.72 - - - - - - - - - - - - - - - - - - - - - - - - -
2012-12-25 09:01:02 KST comment server INFO 200 - Hardware Swap Space: 2047MB/2047MB - - - 0.721 - - - - - - - - - - - - - - - - - - - - - - - - -
2012-12-25 09:01:02 KST comment server INFO 200 - Max File Descriptor Count: 1024 - - - 0.722 - - - - - - - - - - - - - - - - - - - - - - - - -
2012-12-25 09:01:02 KST comment server INFO 200 - Open File Descriptor Count: 44 - - - 0.722 - - - - - - - - - - - - - - - - - - - - - - - - -
2012-12-25 09:01:02 KST comment server INFO 200 - OS Name: Linux - - - 0.723 - - - - - - - - - - - - - - - - - - - - - - - - -
2012-12-25 09:01:02 KST comment server INFO 200 - OS Version: 2.6.18-274.7.1.el5xen - - - 0.724 - - - - - - - - - - - - - - - - - - - - - - - - -
2012-12-25 09:01:02 KST comment server INFO 200 - OS Architecture: amd64 - - - 0.724 - - - - - - - - - - - - - - - - - - - - - - - - -
2012-12-25 09:01:02 KST comment server INFO 200 - Java Name: Java HotSpot(TM) 64-Bit Server VM - - - 0.725 - - - - - - - - - - - - - - - - - - - - - - - - -
2012-12-25 09:01:02 KST comment server INFO 200 - Java Vendor: Oracle Corporation - - - 0.725 - - - - - - - - - - - - - - - - - - - - - - - - -
2012-12-25 09:01:02 KST comment server INFO 200 - Java Version: 1.7.0_03 - - - 0.726 - - - - - - - - - - - - - - - - - - - - - - - - -
2012-12-25 09:01:02 KST comment server INFO 200 - Java VM Version: 22.1-b02 - - - 0.726 - - - - - - - - - - - - - - - - - - - - - - - - -
2012-12-25 09:01:02 KST comment server INFO 200 - Java Spec Version: 1.7 - - - 0.727 - - - - - - - - - - - - - - - - - - - - - - - - -
2012-12-25 09:01:02 KST comment server INFO 200 - Java Home: /usr/java/jdk1.7.0_03/jre - - - 0.728 - - - - - - - - - - - - - - - - - - - - - - - - -
2012-12-25 09:01:02 KST comment server INFO 200 - Java Max Heap Size: 4916MB - - - 0.728 - - - - - - - - - - - - - - - - - - - - - - - - -
2012-12-25 09:01:02 KST comment server INFO 200 - Java Architecture: 64 - - - 0.729 - - - - - - - - - - - - - - - - - - - - - - - - -
2012-12-25 09:01:02 KST comment server INFO 200 - Java Encoding[file.encoding]: UTF-8 - - - 0.729 - - - - - - - - - - - - - - - - - - - - - - - - -
2012-12-25 09:01:02 KST comment server INFO 200 - Java Args[0]: -Xmx5000M - - - 0.73 - - - - - - - - - - - - - - - - - - - - - - - - -
2012-12-25 09:01:02 KST comment server INFO 200 - Java Args[1]: -XX:+UseConcMarkSweepGC - - - 0.731 - - - - - - - - - - - - - - - - - - - - - - - - -
2012-12-25 09:01:02 KST comment server INFO 200 - Java Args[2]: -XX:+CMSIncrementalMode - - - 0.731 - - - - - - - - - - - - - - - - - - - - - - - - -
2012-12-25 09:01:02 KST comment server INFO 200 - Java Args[3]: -XX:+UseParNewGC - - - 0.732 - - - - - - - - - - - - - - - - - - - - - - - - -
2012-12-25 09:01:02 KST comment server INFO 200 - Java Args[4]: -XX:NewSize=256m - - - 0.732 - - - - - - - - - - - - - - - - - - - - - - - - -
2012-12-25 09:01:02 KST comment server INFO 200 - Java Args[5]: -Djava.net.preferIPv4Stack=true - - - 0.733 - - - - - - - - - - - - - - - - - - - - - - - - -
2012-12-25 09:01:02 KST comment server INFO 200 - Java Args[6]: -Dcom.sun.management.jmxremote=true - - - 0.733 - - - - - - - - - - - - - - - - - - - - - - - - -
2012-12-25 09:01:02 KST comment server INFO 200 - Java Args[7]: -Dcom.wowza.wms.AppHome=/WowzaMediaServer - - - 0.734 - - - - - - - - - - - - - - - - - - - - - - - - -
2012-12-25 09:01:02 KST comment server INFO 200 - Java Args[8]: -Dcom.wowza.wms.ConfigURL= - - - 0.734 - - - - - - - - - - - - - - - - - - - - - - - - -
2012-12-25 09:01:02 KST comment server INFO 200 - Java Args[9]: -Dcom.wowza.wms.ConfigHome=/WowzaMediaServer - - - 0.735 - - - - - - - - - - - - - - - - - - - - - - - - -
2012-12-25 09:01:02 KST comment server INFO 200 - Java GC[0]: ParNew - - - 0.735 - - - - - - - - - - - - - - - - - - - - - - - - -
2012-12-25 09:01:02 KST comment server INFO 200 - Java GC[1]: ConcurrentMarkSweep - - - 0.736 - - - - - - - - - - - - - - - - - - - - - - - - -
2012-12-25 09:01:02 KST comment server INFO 200 - Server threads[h/t]: 10/10 - - - 0.737 - - - - - - - - - - - - - - - - - - - - - - - - -
2012-12-25 09:01:04 KST comment server INFO 200 - JMX java.rmi.server.hostname: xxx.xxx.xxx.xxx - - - 2.346 - - - - - - - - - - - - - - - - - - - - - - - - -
2012-12-25 09:01:04 KST comment server INFO 200 - JMX bind attempt service:jmx:rmi://127.0.0.1:8084/jndi/rmi://127.0.0.1:8085/jmxrmi - - - 2.346 - - - - - - - - - - - - - - - - - - - - - - - - -
2012-12-25 09:01:04 KST comment server INFO 200 - JMX password file: /WowzaMediaServer/conf/jmxremote.password - - - 2.35 - - - - - - - - - - - - - - - - - - - - - - - - -
2012-12-25 09:01:04 KST comment server INFO 200 - JMX access file: /WowzaMediaServer/conf/jmxremote.access - - - 2.35 - - - - - - - - - - - - - - - - - - - - - - - - -
2012-12-25 09:01:04 KST comment server INFO 200 - JMX bind successful - - - 2.549 - - - - - - - - - - - - - - - - - - - - - - - - -
2012-12-25 09:01:04 KST comment server INFO 200 - CMDInterface now listening: [any]:8083 - - - 2.598 - - - - - - - - - - - - - - - - - - - - - - - - -
2012-12-25 09:01:05 KST vhost-start vhost INFO 200 _defaultVHost_ - - - - 3.261 - - - - - - - - - - - - - - - - - - - - - - - - -
2012-12-25 09:01:05 KST comment server INFO 200 - _defaultVHost_ threads[h/t]:360/240 home:/WowzaMediaServer - - - 3.325 - - - - - - - - - - - - - - - - - - - - - - - - -
2012-12-25 09:01:05 KST comment vhost INFO 200 _defaultVHost_ Bind attempt ([any]:1935:6) - - - 3.331 - - - - - - - - - - - - - - - - - - - - - - - - -
2012-12-25 09:01:05 KST comment vhost INFO 200 _defaultVHost_ Bind successful ([any]:1935) - - - 3.375 - - - - - - - - - - - - - - - - - - - - - - - - -
2012-12-25 09:01:05 KST comment vhost INFO 200 _defaultVHost_ Bind attempt ([any]:554:6) - - - 3.378 - - - - - - - - - - - - - - - - - - - - - - - - -
2012-12-25 09:01:05 KST comment vhost INFO 200 _defaultVHost_ Bind successful ([any]:554) - - - 3.379 - - - - - - - - - - - - - - - - - - - - - - - - -
2012-12-25 09:01:05 KST comment vhost INFO 200 _defaultVHost_ Bind attempt ([any]:8086:1) - - - 3.379 - - - - - - - - - - - - - - - - - - - - - - - - -
2012-12-25 09:01:05 KST comment vhost INFO 200 _defaultVHost_ Bind successful ([any]:8086) - - - 3.38 - - - - - - - - - - - - - - - - - - - - - - - - -
Richard, thank you for advice.
I reduced java heap size to 4000M as your advice. (option: -Xmx4000M )
There was an exactly same problem after 3 days.
Is there any other hints for this problem?
I will try anything to solve this problem.
Richard, thanks for your advice.
I checked and test again as your advice.
-
Of course this WowzaMediaServer runs as “-server” mode.
-
I removed this GC option, but it still has same problem.
-
I set step #6 for 12 core, such as ProcessorCount, PoolSize. -> It has same problem.
-
I increase RAM from 7G to 11G and set 8G JAVA heap size and I added “NewSize=512” option. -> It has same problem.
-
I don’t try origin/edge cluster not yet. This will be final step in this situation.
I found suspicious option “RTSPSessionTimeout=0” that is in “Application.xml”.
I want to try test as “RTSPSessionTimeout=90000” in “Applicaiont.xml”.
But now some Android apps can’t keep connection without option “RTSPSessionTimeout=0”.
Is it possible “RTSPSessionTimeout=0” can be cause of this memory problem?
So, I tried analyzing java heap memory dump.
Please look around the capture image.
No, it means the /conf/[app-name]/Application.xml /RTP /RTSPSessionTimeout
<RTP>
<AVSyncMethod>senderreport</AVSyncMethod>
<MaxRTCPWaitTime>12000</MaxRTCPWaitTime>
<IdleFrequency>75</IdleFrequency>
[COLOR="#B22222"][B]<RTSPSessionTimeout>0</RTSPSessionTimeout>[/B][/COLOR]
<RTSPMaximumPendingWriteBytes>0</RTSPMaximumPendingWriteBytes>
<RTSPBindIpAddress></RTSPBindIpAddress>
<RTSPConnectionIpAddress>0.0.0.0</RTSPConnectionIpAddress>
<RTSPOriginIpAddress>127.0.0.1</RTSPOriginIpAddress>
<IncomingDatagramPortRanges>*</IncomingDatagramPortRanges>
<!-- Properties defined here will override any properties defined in conf/RTP.xml for any depacketizers loaded by this application -->
<Properties>
</Properties>
</RTP>
Actually, I can’t upgrade Wowza version to 3.5 right now.
And there are no errors in access or error log.
It just freeze.
I’m trying test as “7200000”.
May be I can get the result 1 or 2 days later.
Here is the result of testing with option “RTSPSessionTimeout” in /conf/[app-name]/Application.xml /RTP
Before: 0 (The WowzaMediaServer was freezing in every 1 or 2 days. Only the solution is restarting.)
Now: 7200000
The WowzaMediaServer doesn’t freeze for 8 days and it works stable yet.
It was effective setting in compare with before.
I attached some graph of memory usage
1. WowzaMediaServer freezing pattern.
2. WowzaMediaServer works stable pattern.
I’m not sure the problem has been solved yet. Because of the red box in this graph.
Anyway, It works stable till now.
Please let me know your opinion of this result.