Hi,
Our one of the wowza server was running fine, but now a days it started getting heap memory issues, and cpu utilization was also getting high. Due to this streams are being stopped from streaming and requires a restart.
Earlier wowza was running with about 30 streams , but we enabled pulling from another wowza server (sitting on amazon) for distribution of streams for all of them, after that it started giving memory issues.
Below is the log on wowza o/p.
Configure logging: file:///usr/local/WowzaMediaServer/conf/log4j.properties
INFO server server-start Wowza Media Server 2 Subscription 2.2.4 build27452 -
INFO server comment - Serial number: XXXXX-XXXXX-XXXXX-XXXXX-xxxxx
INFO server comment - Maximum connections: Unlimited
INFO server comment - Hardware Available Processors: 4
INFO server comment - Hardware Physical Memory: 13233MB/16054MB
INFO server comment - Hardware Swap Space: 16386MB/16386MB
INFO server comment - Max File Descriptor Count: 20000
INFO server comment - Open File Descriptor Count: 41
INFO server comment - OS Name: Linux
INFO server comment - OS Version: 2.6.18-194.32.1.el5
INFO server comment - OS Architecture: amd64
INFO server comment - Java Name: Java HotSpot(TM) 64-Bit Server VM
INFO server comment - Java Vendor: Sun Microsystems Inc.
INFO server comment - Java Version: 1.6.0_21
INFO server comment - Java VM Version: 17.0-b16
INFO server comment - Java Spec Version: 1.6
INFO server comment - Java Home: /usr/java/jdk1.6.0_21/jre
INFO server comment - Java Max Heap Size: 3991MB
INFO server comment - Java Architecture: 64
INFO server comment - Java Encoding[file.encoding]: UTF-8
INFO server comment - Java Args[0]: -Xmx4000M
INFO server comment - Java Args[1]: -XX:+UseConcMarkSweepGC
INFO server comment - Java Args[2]: -XX:+CMSIncrementalMode
INFO server comment - Java Args[3]: -XX:+UseParNewGC
INFO server comment - Java Args[4]: -Djava.net.preferIPv4Stack=true
INFO server comment - Java Args[5]: -Dcom.sun.management.jmxremote=true
INFO server comment - Java Args[6]: -Dcom.wowza.wms.AppHome=/usr/local/WowzaMediaServer
INFO server comment - Java Args[7]: -Dcom.wowza.wms.ConfigURL=
INFO server comment - Java Args[8]: -Dcom.wowza.wms.ConfigHome=/usr/local/WowzaMediaServer
INFO server comment - Java GC[0]: ParNew
INFO server comment - Java GC[1]: ConcurrentMarkSweep
INFO server comment - Server threads[h/t]: 10/10
INFO server comment - JMX java.rmi.server.hostname: WOWZA_IP
INFO server comment - JMX bind attempt service:jmx:rmi://WOWZA_IP:8084/jndi/rmi://WOWZA_IP:8085/jmxrmi
INFO server comment - JMX password file: /usr/local/WowzaMediaServer/conf/jmxremote.password
INFO server comment - JMX access file: /usr/local/WowzaMediaServer/conf/jmxremote.access
INFO server comment - JMX bind successful
INFO server comment - CMDInterface now listening: [any]:8083
INFO vhost vhost-start _defaultVHost_ -
[ DELLSERVER_WOWZA_EXPAT root@localhost.localdomain:/usr/local/WowzaMediaServer/WOWZALOGS]
Below is the setenv.sh, I have increased heap memory to 4GB instead of 3Gb, still the issue persists.
#!/bin/sh
_EXECJAVA=java
#JAVA_OPTS="-Xmx3000M"
JAVA_OPTS="-Xmx4000M"
# Uncomment to run server environment (faster), Note: will only work if server VM install, comes with JDL
JAVA_OPTS="$JAVA_OPTS -server"
# Can be a better GC setting to avoid long pauses
JAVA_OPTS="$JAVA_OPTS -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:+UseParNewGC"
# Uncomment to fix multicast crosstalk problem when streams share multicast port
JAVA_OPTS="$JAVA_OPTS -Djava.net.preferIPv4Stack=true"
WMSAPP_HOME=/usr/local/WowzaMediaServer
WMSCONFIG_HOME=/usr/local/WowzaMediaServer
WMSCONFIG_URL=
export WMSAPP_HOME WMSCONFIG_HOME JAVA_OPTS _EXECJAVA
wms.sh has below lines,
#!/bin/sh
# NOTE: Here you can configure the JVM's built in JMX interface.
# See the "Server Management Console and Monitoring" chapter
# of the "User's Guide" for more information on how to configure the
# remote JMX interface in the [install-dir]/conf/Server.xml file.
JMXOPTIONS=-Dcom.sun.management.jmxremote=true
#JMXOPTIONS="$JMXOPTIONS -Dcom.sun.management.jmxremote.port=1099"
#JMXOPTIONS="$JMXOPTIONS -Dcom.sun.management.jmxremote.authenticate=true"
#JMXOPTIONS="$JMXOPTIONS -Dcom.sun.management.jmxremote.ssl=false"
#JMXOPTIONS="$JMXOPTIONS -Dcom.sun.management.jmxremote.password.file=$WMSCONFIG_HOME/conf/jmxremote.password"
#JMXOPTIONS="$JMXOPTIONS -Dcom.sun.management.jmxremote.access.file=$WMSCONFIG_HOME/conf/jmxremote.access"
#JMXOPTIONS="$JMXOPTIONS -Djava.rmi.server.hostname=192.168.1.7"
if [ "$WMSCOMMAND" != "start" ]; then
JMXOPTIONS=""
fi
ulimit -n 20000
# log interceptor com.wowza.wms.logging.LogNotify - see Javadocs for ILogNotify
$_EXECJAVA $JAVA_OPTS $JMXOPTIONS -Dcom.wowza.wms.AppHome="$WMSAPP_HOME" -Dcom.wowza.wms.ConfigURL="$WMSCONFIG_URL" -Dcom.wowza.wms.ConfigHome="$WMSCONFIG_HOME" -cp $WMSAPP_HOME/bin/wms-bootstrap.jar com.wowza.wms.bootstrap.Bootstrap $WMSCOMMAND >/dev/null 2>&1 &
if [ "$WMSCOMMAND" = "start" ]; then
echo $! > ${WMSPIDFILE}
wait $!
fi
exit 0
Server is of 16GB, only wowza is running there, does the 4GB is enough for heap (mentioned in sevenv.sh) or it needs to be increased?
Here is the Image of memory usage and cpu utilization
IMAGE/
can anybody point out where the issues are present.
Thanks,
Shekar
Hi shekarkcb,
Is the screenshot showing your Wowza when it is serving 30 streams?
Can you provide logs or console output at full load when the streams are getting dropped?
Review the tuning guide, specifically step 6. I would take out the GC lines, add the -server flag
https://www.wowza.com/docs/how-to-do-performance-tuning
You could afford to raise Java heap size. Try -Xmx6000M
Richard
Right, comment that out. But add -server flag to utilize JDK built-in GC. The advice from step 6 is to try this special GC (or another option) if there are problems, but if there are not problems it might cause problems.
I suggested going to 6000M heap size because you have plenty of memory, you can. 4000m is a lot too. Maybe the above change will help.
Richard
Monitor that with JConsole. Let us know if that helps
Richard
Hi,
I shekarkcb, did you resolve this issue?
I have similar issue in other environment.
But, in my case,
java process utilizes 2G memory
java heap utilizes 100M memory
I use jconsole and jmap and top for memory monitoring.
jmap -histo 1539 > histo
jmap -heap 1539 > heap
where 1539 is my pid.
could you share your results for jmap and top commands under 30 streams load?
Thanks for the reply. Yes the screen shot is @ 30 streams. Logs are not saved currently, will upload them once i got them back. Just wanted to find what is causing memory issue?
I mean when the memory set to 3GB, wouldn’t that enough to server 30+ streams? When we introduced another wowza server on amazon which is pulling all the streams from this wowza issues are started arising.
Thanks,
ShekarKCB
Below is the updated log.
#Version: 1.0
#Start-Date: 2011-09-15 00:15:26 IST
#Software: Wowza Media Server 2.2.4 build27452
#Date: 2011-09-15
#Fields: x-severity x-category x-event date time c-client-id c-ip c-port cs-bytes sc-bytes x-duration x-sname x-stream-id x-spos sc-stream-bytes cs-stream-bytes x-file-size x-file-length x-ctx x-comment
WARN server comment 2011-09-15 00:15:26 - - - - - 9298.693 - - - - - - -- LiveStreamPacketizerSmoothStreaming.handlePacket[feed016/_definst_/feed016.stream]: Fragment duration greater than suggested range of 1-4 seconds. Adjust keyframe interval accordingly: Fragment durations: [5.7,4.8,4.8]
WARN server comment 2011-09-15 00:15:26 - - - - - 9298.723 - - - - - - -- LiveStreamPacketizerSmoothStreaming.handlePacket[feed020/_definst_/feed020.stream]: Fragment duration greater than suggested range of 1-4 seconds. Adjust keyframe interval accordingly: Fragment durations: [4.8,3.8,5.7]
WARN server comment 2011-09-15 00:15:26 - - - - - 9298.732 - - - - - - -- LiveStreamPacketizerSmoothStreaming.handlePacket[feed029/_definst_/feed029.stream]: Fragment duration greater than suggested range of 1-4 seconds. Adjust keyframe interval accordingly: Fragment durations: [4.8,4.8,5.7]
WARN server comment 2011-09-15 00:17:38 - - - - - 9430.557 - - - - - - -- LiveStreamPacketizerSmoothStreaming.handlePacket[feed031/_definst_/feed031.stream]: Fragment duration greater than suggested range of 1-4 seconds. Adjust keyframe interval accordingly: Fragment durations: [4.8,3.8,0.9]
WARN server comment 2011-09-15 00:17:38 - - - - - 9430.605 - - - - - - -- LiveStreamPacketizerSmoothStreaming.handlePacket[feed05/_definst_/feed05.stream]: Fragment duration greater than suggested range of 1-4 seconds. Adjust keyframe interval accordingly: Fragment durations: [4.8,4.8,4.8]
LOGS AFTER RESTARTING THE WOWZA SERVER:
======================================
#Version: 1.0
#Start-Date: 2011-09-15 06:09:37 IST
#Software: Wowza Media Server 2.2.4 build27452
#Date: 2011-09-15
#Fields: x-severity x-category x-event date time c-client-id c-ip c-port cs-bytes sc-bytes x-duration x-sname x-stream-id x-spos sc-stream-bytes cs-stream-bytes x-file-size x-file-length x-ctx x-comment
WARN server comment 2011-09-15 06:09:43 - - - - - 6.12 - - - - - - - - Failed to connect: org.apache.mina.common.RuntimeIOException: Failed to get the session[ConnectException: Connection refused].
WARN server comment 2011-09-15 06:09:43 - - - - - 6.134 - - - - - - - - Failed to connect: org.apache.mina.common.RuntimeIOException: Failed to get the session[ConnectException: Connection refused].
WARN server comment 2011-09-15 06:09:43 - - - - - 6.15 - - - - - - - - Failed to connect: org.apache.mina.common.RuntimeIOException: Failed to get the session[ConnectException: Connection refused].
WARN server comment 2011-09-15 06:09:46 - - - - - 9.12 - - - - - - - - Failed to connect: org.apache.mina.common.RuntimeIOException: Failed to get the session[ConnectException: Connection refused].
WARN server comment 2011-09-15 06:09:46 - - - - - 9.137 - - - - - - - - Failed to connect: org.apache.mina.common.RuntimeIOException: Failed to get the session[ConnectException: Connection refused].
WARN server comment 2011-09-15 06:09:46 - - - - - 9.151 - - - - - - - - Failed to connect: org.apache.mina.common.RuntimeIOException: Failed to get the session[ConnectException: Connection refused].
WARN server comment 2011-09-15 06:09:49 - - - - - 12.121 - - - - - - - - Failed to connect: org.apache.mina.common.RuntimeIOException: Failed to get the session[ConnectException: Connection refused].
WARN server comment 2011-09-15 06:09:49 - - - - - 12.138 - - - - - - - - Failed to connect: org.apache.mina.common.RuntimeIOException: Failed to get the session[ConnectException: Connection refused].
WARN server comment 2011-09-15 06:09:49 - - - - - 12.153 - - - - - - - - Failed to connect: org.apache.mina.common.RuntimeIOException: Failed to get the session[ConnectException: Connection refused].
WARN server comment 2011-09-15 06:09:49 - - - - - 12.225 - - - - - - - - Failed to connect: org.apache.mina.common.RuntimeIOException: Failed to get the session[ConnectException: Connection refused].
WARN server comment 2011-09-15 06:09:49 - - - - - 12.245 - - - - - - - - Failed to connect: org.apache.mina.common.RuntimeIOException: Failed to get the session[ConnectException: Connection refused].
WARN server comment 2011-09-15 06:09:49 - - - - - 12.257 - - - - - - - - Failed to connect: org.apache.mina.common.RuntimeIOException: Failed to get the session[ConnectException: Connection refused].
WARN server comment 2011-09-15 06:09:52 - - - - - 15.229 - - - - - - - - Failed to connect: org.apache.mina.common.RuntimeIOException: Failed to get the session[ConnectException: Connection refused].
WARN server comment 2011-09-15 06:09:52 - - - - - 15.245 - - - - - - - - Failed to connect: org.apache.mina.common.RuntimeIOException: Failed to get the session[ConnectException: Connection refused].
WARN server comment 2011-09-15 06:09:52 - - - - - 15.274 - - - - - - - - Failed to connect: org.apache.mina.common.RuntimeIOException: Failed to get the session[ConnectException: Connection refused].
WARN server comment 2011-09-15 06:09:55 - - - - - 18.229 - - - - - - - - Failed to connect: org.apache.mina.common.RuntimeIOException: Failed to get the session[ConnectException: Connection refused].
WARN server comment 2011-09-15 06:09:55 - - - - - 18.247 - - - - -
- - LiveStreamPacketizerSmoothStreaming.handlePacket[feed032/_definst_/feed032.stream]: Fragment duration greater than suggested range of 1-4 seconds. Adjust keyframe interval accordingly: Fragment durations: [3.8,4.8,4.8]
WARN server comment 2011-09-15 08:25:18 - - - - - 8141.708 - - - - - - - - LiveStreamPacketizerSmoothStreaming.handlePacket[feed029/_definst_/feed029.stream]: Fragment duration greater than suggested range of 1-4 seconds. Adjust keyframe interval accordingly: Fragment durations: [3.8,9.6,4.8]
WARN server comment 2011-09-15 08:27:36 - - - - - 8279.348 - - - - - - - - LiveStreamPacketizerSmoothStreaming.handlePacket[feed05/_definst_/feed05.stream]: Fragment duration greater than suggested range of 1-4 seconds. Adjust keyframe interval accordingly: Fragment durations: [4.8,4.8,4.8]
WARN server comment 2011-09-15 08:27:40 - - - - - 8283.741 - - - - - - - - LiveStreamPacketizerSmoothStreaming.handlePacket[feed011/_definst_/feed011.stream]: Fragment duration greater than suggested range of 1-4 seconds. Adjust keyframe interval accordingly: Fragment durations: [4.8,4.8,3.8]
WARN server comment 2011-09-15 08:28:53 - - - - - 8356.335 - - - - - - - - LiveStreamPacketizerSmoothStreaming.handlePacket[feed030/_definst_/feed030.stream]: Fragment duration greater than suggested range of 1-4 seconds. Adjust keyframe interval accordingly: Fragment durations: [9.6,4.8,9.6]
WARN server comment 2011-09-15 08:28:53 - - - - - 8356.341 - - - - - - - - LiveStreamPacketizerSmoothStreaming.handlePacket[feed025/_definst_/feed025.stream]: Fragment duration greater than suggested range of 1-4 seconds. Adjust keyframe interval accordingly: Fragment durations: [3.8,0.9,4.8]
WARN server comment 2011-09-15 08:28:53 -
WOWZA ACCESS LOGS:
=================
INFO server comment - PushPublisher#PingResult.onResult[rtmp://cdn-host:1935/rtmp/push/loginid/secure/live/feed040]: false
INFO server comment - PushPublisher#PingResult.onResult[rtmp://cdn-host:1935/rtmp/push/loginid/secure/live/feed011]: false
INFO server comment - PushPublisher#PingResult.onResult[rtmp://cdn-host:1935/rtmp/push/loginid/secure/live/feed012]: false
INFO server comment - PushPublisher#PingResult.onResult[rtmp://cdn-host:1935/rtmp/push/loginid/secure/live/feed034]: false
INFO server comment - PushPublisher#PingResult.onResult[rtmp://cdn-host:1935/rtmp/push/loginid/secure/live/feed010]: false
INFO server comment - PushPublisher#PingResult.onResult[rtmp://cdn-host:1935/rtmp/push/loginid/secure/live/feed025]: false
INFO server comment - PushPublisher#PingResult.onResult[rtmp://cdn-host:1935/rtmp/push/loginid/secure/live/feed030]: false
INFO server comment - PushPublisher#PingResult.onResult[rtmp://cdn-host:1935/rtmp/push/loginid/secure/live/feed013]: false
INFO server comment - PushPublisher#PingResult.onResult[rtmp://cdn-host:1935/rtmp/push/loginid/live/feed036]: false
INFO server comment - PushPublisher#PingResult.onResult[rtmp://cdn-host:1935/rtmp/push/loginid/secure/live/feed040]: false
INFO server comment - PushPublisher#PingResult.onResult[rtmp://cdn-host:1935/rtmp/push/loginid/secure/live/feed034]: false
INFO server comment - PushPublisher#PingResult.onResult[rtmp://cdn-host:1935/rtmp/push/loginid/secure/live/feed010]: false
INFO server comment - PushPublisher#PingResult.onResult[rtmp://cdn-host:1935/rtmp/push/loginid/secure/live/feed025]: false
INFO server comment - PushPublisher#PingResult.onResult[rtmp://cdn-host:1935/rtmp/push/loginid/secure/live/feed030]: false
INFO server comment - PushPublisher#PingResult.onResult[rtmp://cdn-host:1935/rtmp/push/loginid/secure/live/feed013]: false
INFO server comment - PushPublisher#PingResult.onResult[rtmp://cdn-host:1935/rtmp/push/loginid/secure/live/feed011]: false
INFO server comment - PushPublisher#PingResult.onResult[rtmp://cdn-host:1935/rtmp/push/loginid/secure/live/feed012]: false
INFO server comment - PushPublisher#PingResult.onResult[rtmp://cdn-host:1935/rtmp/push/loginid/live/feed036]: false
INFO server comment - PushPublisher#PingResult.onResult[rtmp://cdn-host:1935/rtmp/push/loginid/secure/live/feed040]: false
INFO server comment - PushPublisher#PingResult.onResult[rtmp://cdn-host:1935/rtmp/push/loginid/secure/live/feed011]: false
INFO server comment - PushPublisher#PingResult.onResult[rtmp://cdn-host:1935/rtmp/push/loginid/secure/live/feed012]: false
INFO server comment - PushPublisher#PingResult.onResult[rtmp://cdn-host:1935/rtmp/push/loginid/secure/live/feed034]: false
INFO server comment - PushPublisher#PingResult.onResult[rtmp://cdn-host:1935/rtmp/push/loginid/secure/live/feed010]: false
INFO server comment - PushPublisher#PingResult.onResult[rtmp://cdn-host:1935/rtmp/push/loginid/secure/live/feed025]: false
INFO server comment - PushPublisher#PingResult.onResult[rtmp://cdn-host:1935/rtmp/push/loginid/secure/live/feed030]: false
INFO server comment - PushPublisher#PingResult.onResult[rtmp://cdn-host:1935/rtmp/push/loginid/secure/live/feed013]: false
INFO server comment - PushPublisher#PingResult.onResult[rtmp://cdn-host:1935/rtmp/push/loginid/secure/live/feed040]: false
INFO server comment - PushPublisher#PingResult.onResult[rtmp://cdn-host:1935/rtmp/push/loginid/live/feed036]: false
INFO server comment - CommandInterfaceCommandShutdownServer.invoke: Shutting down server.
INFO server comment - CommandInterfaceCommandShutdownServer.invoke: Shutting down server.
startup.sh: line 24: 13745 Killed $_EXECJAVA $JAVA_OPTS $JMXOPTIONS -Dcom.wowza.wms.AppHome="$WMSAPP_HOME" -Dcom.wowza.wms.ConfigURL="$WMSCONFIG_URL" -Dcom.wowza.wms.ConfigHome="$WMSCONFIG_HOME" -cp $WMSAPP_HOME/bin/wms-bootstrap.jar com.wowza.wms.bootstrap.Bootstrap start
Here is the another screen shot.
Thanks,
ShekarKCB
Update 2 -
Once the pull from Amazon wowza server is stopped, memory came down from 90% to 45%… !!!
Thanks,
ShekarKCB
Thanks for the reply.
But key frame issue shouldn’t cause memory to be high/cpu utilization high. As you see the logs, it complains key frame interval issue, and says stream may not play on Ipad/iphone tats okay for me.
Do you think key frame settings is making noise here?
Thanks,
ShekarKCB
Thanks for the reply Richard.
Do you want me to comment the line
# Can be a better GC setting to avoid long pauses
JAVA_OPTS="$JAVA_OPTS -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:+UseParNewGC"
I have increaed the Java heap size from 4G to 6G as you have mentioned, any logic for this?
Just wanted to know, Does this depends on number of streams being streamed ( and to be increased when in addition to streaming some other server also pulls from this server? ), as we don’t wanted to disturb setups running on production servers …!!!
Thanks,
Shekar
Thanks for the reply Richard.
I think -server is already added by default, here is the updated setenv.sh file, pls let me know this is good enough to overcome the issue.
#!/bin/sh
_EXECJAVA=java
#JAVA_OPTS="-Xmx3000M"
#JAVA_OPTS="-Xmx4000M"
#Updated to 6G as per wowza forum suggestion to tackle heap mem issue
JAVA_OPTS="-Xmx6000M"
# Uncomment to run server environment (faster), Note: will only work if server VM install, comes with JDL
JAVA_OPTS="$JAVA_OPTS -server"
#Commenting Now, sep 16, to tackle heap mem issue
# Can be a better GC setting to avoid long pauses
#JAVA_OPTS="$JAVA_OPTS -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:+UseParNewGC"
# Uncomment to fix multicast crosstalk problem when streams share multicast port
JAVA_OPTS="$JAVA_OPTS -Djava.net.preferIPv4Stack=true"
WMSAPP_HOME=/usr/local/WowzaMediaServer
WMSCONFIG_HOME=/usr/local/WowzaMediaServer
WMSCONFIG_URL=
export WMSAPP_HOME WMSCONFIG_HOME JAVA_OPTS _EXECJAVA
Thanks,
ShekarKCB
Thanks for the reply.
Will apply the new changes, and let you know issues if any.
Thanks,
ShekarKCB
It mentions a keyframe problem. Have you tried setting your keyframe interval to 2 seconds?
