stream repeater causing java.lang.OutOfMemoryError: Java heap space and carshing

Wowza Streaming Engine
1

hi

we have a scenario where configured repeated streams(pulled) are down due to “no service” , the Wowza instance retry to pull those streams till he crash.

the first warning is :LiveMediaStreamReceiver.doWatchdog: streamTimeout: Resetting connection

after many retries with the above warning we get : Unexpected exception.java.lang.OutOfMemoryError: Java heap space

java.lang.OutOfMemoryError: Java heap space

at java.nio.HeapByteBuffer.(Unknown Source)

at java.nio.ByteBuffer.allocate(Unknown Source)

at org.apache.mina.common.SimpleByteBufferAllocator.allocate(Unknown Source)

at org.apache.mina.common.ByteBuffer.allocate(Unknown Source)

at org.apache.mina.transport.socket.nio.SocketIoProcessor.read(Unknown Source)

at org.apache.mina.transport.socket.nio.SocketIoProcessor.process(Unknown Source)

at org.apache.mina.transport.socket.nio.SocketIoProcessor.access$600(Unknown Source)

at org.apache.mina.transport.socket.nio.SocketIoProcessor$Worker.run(Unknown Source)

at org.apache.mina.util.NamePreservingRunnable.run(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

at java.lang.Thread.run(Unknown Source)

another warnings on this scenario are : NetConnectionConnection.connect: Failed to connect[xxx.xx.xxxx.net:1935]: org.apache.mina.common.RuntimeIOException: Failed to get the session[java.net.SocketException: Network is unreachable].

LiveMediaStreamReceiver.internalConnect: [[xxx.xx.xxxx.net:1935]:java.nio.channels.UnresolvedAddressException

the current setenv.sh settings

#!/bin/sh

_EXECJAVA=/opt/local/lvs/jre1.7.0_09/bin/java

JAVA_OPTS="-Xmx3000M"

Uncomment to run server environment (faster), Note: will only work if server VM install, comes with JDL

JAVA_OPTS="$JAVA_OPTS -server"

Can be a better GC setting to avoid long pauses

#JAVA_OPTS="$JAVA_OPTS -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:NewSize=512m"

Uncomment to log debug garbage collection information

#NOW=$(date +"%Y-%m-%d-%H-%M")

#JAVA_OPTS="$JAVA_OPTS -verbose:gc -Xloggc:"/usr/local/WowzaMediaServer/logs/gc_$NOW.log" -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintHeapAtGC"

Uncomment to write heap dump on OutOfMemoryError

#JAVA_OPTS="$JAVA_OPTS -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/usr/local/WowzaMediaServer"

Uncomment to force Java to use specific language settings

#JAVA_OPTS="$JAVA_OPTS -Duser.language=en -Duser.country=US -Dfile.encoding=Cp1252"

Uncomment to fix multicast crosstalk problem when streams share multicast port

JAVA_OPTS="$JAVA_OPTS -Djava.net.preferIPv4Stack=true"

WMSAPP_HOME=/opt/local/lvs/WowzaMediaServer

WMSCONFIG_HOME=/opt/local/lvs/WowzaMediaServer

WMSCONFIG_URL=

export WMSAPP_HOME WMSCONFIG_HOME JAVA_OPTS _EXECJAVA

How can we avoid this from happening ?

Yaniv

2

Hi there, these messages are due to network problems.

I am not sure if the outOfMemory issue is due to network problems. Do you have any custom modules enabled?

Can you post your start-up messages?

Salvadore

3

Hi

true those messages are due to network problems , we fail to pull stream since its temporary down.

when we press “stop receiving streams” the warning are gone.

when streams are up we get no warning and no crashes.

ether way Wowza shouldn’t crash or get out of memory due to failure on pulling streams.

Yaniv

4

Hi Salvadore

below is the startup print , can u share what u looking for ?

comment server INFO 200 - Server License Key: SVRM3-XXXXX-XXXXX-XXXXX-XXXXX-aCQDz - - - 0.647 - - - - - - - - - - - - - - - - - - - - - - - - -

comment server INFO 200 - Maximum Connections: Unlimited - - - 0.649 - - - - - - - - - - - - - - - - - - - - - - - - -

comment server INFO 200 - Transcoder Streams Available: Unlimited - - - 0.654 - - - - - - - - - - - - - - - - - - - - - - - - -

comment server INFO 200 - Transcoder Watermark: No - - - 0.655 - - - - - - - - - - - - - - - - - - - - - - - - -

comment server INFO 200 - nDVR Available: Yes - - - 0.655 - - - - - - - - - - - - - - - - - - - - - - - - -

comment server INFO 200 - DRM Available: Yes - - - 0.656 - - - - - - - - - - - - - - - - - - - - - - - - -

comment server INFO 200 - Hardware Available Processors: 4 - - - 0.657 - - - - - - - - - - - - - - - - - - - - - - - - -

comment server INFO 200 - Hardware Physical Memory: 3247MB/3947MB - - - 0.658 - - - - - - - - - - - - - - - - - - - - - - - - -

comment server INFO 200 - Hardware Swap Space: 511MB/511MB - - - 0.658 - - - - - - - - - - - - - - - - - - - - - - - - -

comment server INFO 200 - Max File Descriptor Count: 20000 - - - 0.659 - - - - - - - - - - - - - - - - - - - - - - - - -

comment server INFO 200 - Open File Descriptor Count: 47 - - - 0.66 - - - - - - - - - - - - - - - - - - - - - - - - -

comment server INFO 200 - OS Name: Linux - - - 0.66 - - - - - - - - - - - - - - - - - - - - - - - - -

comment server INFO 200 - OS Version: 2.6.18-308.16.1.el5 - - - 0.661 - - - - - - - - - - - - - - - - - - - - - - - - -

comment server INFO 200 - OS Architecture: amd64 - - - 0.662 - - - - - - - - - - - - - - - - - - - - - - - - -

comment server INFO 200 - Java Name: Java HotSpot™ 64-Bit Server VM - - - 0.662 - - - - - - - - - - - - - - - - - - - - - - - - -

comment server INFO 200 - Java Vendor: Oracle Corporation - - - 0.663 - - - - - - - - - - - - - - - - - - - - - - - - -

comment server INFO 200 - Java Version: 1.7.0_09 - - - 0.663 - - - - - - - - - - - - - - - - - - - - - - - - -

comment server INFO 200 - Java VM Version: 23.5-b02 - - - 0.664 - - - - - - - - - - - - - - - - - - - - - - - - -

comment server INFO 200 - Java Spec Version: 1.7 - - - 0.665 - - - - - - - - - - - - - - - - - - - - - - - - -

comment server INFO 200 - Java Home: /opt/local/lvs/jre1.7.0_09 - - - 0.665 - - - - - - - - - - - - - - - - - - - - - - - - -

comment server INFO 200 - Java Max Heap Size: 2666MB - - - 0.666 - - - - - - - - - - - - - - - - - - - - - - - - -

comment server INFO 200 - Java Architecture: 64 - - - 0.666 - - - - - - - - - - - - - - - - - - - - - - - - -

comment server INFO 200 - Java Locale[user.language]: en - - - 0.667 - - - - - - - - - - - - - - - - - - - - - - - - -

comment server INFO 200 - Java Locale[user.country]: US - - - 0.667 - - - - - - - - - - - - - - - - - - - - - - - - -

comment server INFO 200 - Java Locale[file.encoding]: UTF-8 - - - 0.668 - - - - - - - - - - - - - - - - - - - - - - - - -

comment server INFO 200 - Java Timezone[user.timezone]: Etc/GMT - - - 0.669 - - - - - - - - - - - - - - - - - - - - - - - - -

comment server INFO 200 - Java Args[0]: -Xmx3000M - - - 0.669 - - - - - - - - - - - - - - - - - - - - - - - - -

comment server INFO 200 - Java Args[1]: -Djava.net.preferIPv4Stack=true - - - 0.67 - - - - - - - - - - - - - - - - - - - - - - - - -

comment server INFO 200 - Java Args[2]: -Dcom.sun.management.jmxremote=true - - - 0.671 - - - - - - - - - - - - - - - - -

comment server INFO 200 - Java Args[3]: -Dcom.wowza.wms.runmode=service - - - 0.671 - - - - - - - - - - - - - - - - - -

comment server INFO 200 - Java Args[4]: -Dcom.wowza.wms.native.base=linux - - - 0.672 - - - - - - - - - - - - - - - - - -

comment server INFO 200 - Java Args[5]: -Dcom.wowza.wms.AppHome=/opt/local/lvs/WowzaMediaServer - - - 0.672 - - - - - - - - - - - - - - -

comment server INFO 200 - Java Args[6]: -Dcom.wowza.wms.ConfigURL= - - - 0.673 - - - - - - - - - - - - - - - - - -

comment server INFO 200 - Java Args[7]: -Dcom.wowza.wms.ConfigHome=/opt/local/lvs/WowzaMediaServer - - - 0.673 - - - - - - - - - - - - - -

comment server INFO 200 - Server runmode: service - - - 0.674 - - - - - - - - - - - - - - - - - - - - -

comment server INFO 200 - Server native.platform: linux - - - 0.675 - - - - - - - - - - - - - - - - - - - -

comment server INFO 200 - Server threads[h/t]: 10/10 - - - 0.675 - - - - - - - - - - - - - - - - - - - -

comment server INFO 200 - JMX java.rmi.server.hostname: localhost/ - - - 1.824 - - - - - - - - - - - - - - - - - -

comment server INFO 200 - JMX bind attempt service:jmx:rmi://localhost/:8084/jndi/rmi://localhost/:8085/jmxrmi - - - 1.824 - - - - - - - - - - - - -

comment server INFO 200 - JMX password file: /opt/local/lvs/WowzaMediaServer/conf/jmxremote.password - - - 1.827 - - - - - - - - - - - - - -

comment server INFO 200 - JMX access file: /opt/local/lvs/WowzaMediaServer/conf/jmxremote.access - - - 1.828 - - - - - - - - - - - - - - -

comment server INFO 200 - JMX bind successful - - - 1.904 - - - - - - - - - - - - - - - - - - - - -

comment server INFO 200 - CMDInterface now listening: [any]:8083 - - - 1.939 - - - - - - - - - - - - - - - - - - -

vhost-start vhost INFO 200 defaultVHost - - - - 2.693 - - - - - - - - - - - - - - - - - - - - -

comment server INFO 200 - defaultVHost threads[h/t]:120/80 home:/opt/local/lvs/WowzaMediaServer - - - 2.759 - - - - - - - - - - - - - - -

comment vhost INFO 200 defaultVHost Bind attempt ([any]:2255:4) - - - 2.764 - - - - - - - - - - - - - - - - - - -

comment vhost INFO 200 defaultVHost Bind successful ([any]:2255) - - - 2.77 - - - - - - - - - - - - - - - - - - -

comment vhost INFO 200 defaultVHost Bind attempt ([any]:8080:4) - - - 2.773 - - - - - - - - - - - - - - - - - - -

comment vhost INFO 200 defaultVHost Bind successful ([any]:8080) - - - 2.775 - - - - - - - - - - - - - - - - - - -

comment vhost INFO 200 defaultVHost Bind attempt ([any]:1935:4) - - - 2.776 - - - - - - - - - - - - - - - - - - -

comment vhost INFO 200 defaultVHost Bind successful ([any]:1935) - - - 2.777 - - - - - - - - - - - - - - - - - - -

comment vhost INFO 200 defaultVHost Bind attempt ([any]:8086:1) - - - 2.778 - - - - - - - - - - - - - - - - - - -

comment vhost INFO 200 defaultVHost Bind successful ([any]:8086) - - - 2.778 - - - - - - - - - - - - - - - - - - -

comment server INFO 200 - ModuleRTMPAuthenticate.onAppStart: SecureToken is off - - - 2.945 - - - - - - - - - - - - - - - - -

comment server INFO 200 - ModuleRTMPAuthenticate.onAppStart: Authorization password file: /opt/local/lvs/WowzaMediaServer/conf/publish.password - - - 2.945 - - - - - - - - -

app-start application INFO 200 definst secureliveHLS/definst - - - 2.991 - - - - - - - - - - - - - - - - - -

create stream INFO 200 - - - - - 0.011 - - - - - - - - - - 1 0 0 0 - - - - - - - - -

comment server INFO 200 - LiveMediaStreamReceiver.connect: rtmp://xxxxxxxxxxxxxxxxxx_mobileHLS_480p_Opal_7_seats_Baccarat:rtmp://xxxxxxxxxxxxxxxxxx:1935/eurostech[854x480_mobileHLS_480p_Opal_7_seats_Baccarat] - - - 3.049 -

comment server INFO 200 - VHost.startStartupStreams: Stream successfully started [secureliveHLS/definst]: flv:854x480_mobileHLS_480p_Opal_7_seats_Baccarat.stream - - - 3.058 - - - - - -

create stream INFO 200 - - - - - 0.0 - - - - - - - - - - 2 0 0 0 - - - - - - - - -

comment server INFO 200 - LiveMediaStreamReceiver.connect: rtmp://xxxxxxxxxxxxxxxxxx_mobileHLS_480p_Topaz_mini_Baccarat:rtmp://xxxxxxxxxxxxxxxxxx:1935/eurostech[854x480_mobileHLS_480p_Topaz_mini_Baccarat] - - - 3.06 -

comment server INFO 200 - VHost.startStartupStreams: Stream successfully started [secureliveHLS/definst]: flv:854x480_mobileHLS_480p_Topaz_mini_Baccarat.stream - - - 3.062 - - - - - - -

create stream INFO 200 - - - - - 0.0 - - - - - - - - - - 3 0 0 0 - - - - - - - - -

comment server INFO 200 - LiveMediaStreamReceiver.connect: rtmp://xxxxxxxxxxxxxxxxxx_mobileHLS_480p_Diamond_Roulette:rtmp://xxxxxxxxxxxxxxxxxx:1935/eurostech[854x482_mobileHLS_480p_Diamond_Roulette] - - - 3.063 - -

comment server INFO 200 - VHost.startStartupStreams: Stream successfully started [secureliveHLS/definst]: flv:854x482_mobileHLS_480p_Diamond_Roulette.stream - - - 3.065 - - - - - - -

create stream INFO 200 - - - - - 0.0 - - - - - - - - - - 4 0 0 0 - - - - - - - - -

comment server INFO 200 - LiveMediaStreamReceiver.connect: rtmp://xxxxxxxxxxxxxxxxxx_mobileHLS_480p_Pearl_Roulette:rtmp://xxxxxxxxxxxxxxxxxx:1935/eurostech[854x482_mobileHLS_480p_Pearl_Roulette] - - - 3.066 - -

comment server INFO 200 - VHost.startStartupStreams: Stream successfully started [secureliveHLS/definst]: flv:854x482_mobileHLS_480p_Pearl_Roulette.stream - - - 3.068 - - - - - - -

comment server INFO 200 - ModuleRTMPAuthenticate.onAppStart: SecureToken is off - - - 3.127 - - - - - - - - - - - - - - - - -

comment server INFO 200 - ModuleRTMPAuthenticate.onAppStart: Authorization password file: /opt/local/lvs/WowzaMediaServer/conf/publish.password - - - 3.127 - - - - - - - - -

app-start application INFO 200 definst liveHLS/definst - - - 3.133 - - - - - - - - - - - - - - - - - -

create stream INFO 200 - - - - - 0.0 - - - - - - - - - - 1 0 0 0 - - - - - - - - -

comment server INFO 200 - LiveMediaStreamReceiver.connect: rtmp://xxxxxxxxxxxxxxxxxxs_mobileHLS_480p_1495:rtmp://xxxxxxxxxxxxxxxxxx[s_mobileHLS_480p_1495] - - - 3.135 - - - - - - -

comment server INFO 200 - VHost.startStartupStreams: Stream successfully started [liveHLS/definst]: flv:mobileHLS_480p_1495.stream - - - 3.136 - - - - - - - - - -

comment server INFO 200 - Server.startShutdownHook: Start server shutdown hook - - - 3.137 - - - - - - - - - - - - - - - - -

comment server INFO 200 - Wowza Media Server is started! - - - 3.138 - - - - - - - - - - - - - - - - - - - -

comment server INFO 200 -

5

Hi Salvadore thanks for u’r replay

we have modified this value lately due to the crashes we have experienced , it was the default value till 8 days ago.

my problem is still , if stream repeater have values that fail to pull streams my instance will crash after x hours.

u can try it your self … config some dummy/non-existed pulled streams and see your instance go out of memory , more streams that are failing the quicker the memory will run out.

Yaniv

6

10x will try and update …

UPDATE : well the above syntax generates an error.

ERROR server comment 2013-07-08 07:18:18 - - - - - 430709.857 - - - - - - - - Invalid LiveMediaStream URL[{]: java.net.URISyntaxException: Illegal character in path at index 0: {

WARN server comment 2013-07-08 07:18:18 - - - - - 430709.858 - - - - - - - - LiveMediaStreamReceiver.connect: Failure no URL: {

Yaniv

7

Hi Yaniv,

Can you show the startup lines from your access log?

Salvadore

8

I see that your Hardware Physical Memory is: 3247MB/3947MB

With a Java Heap size of 3000MB, which is almost all of your available ram (3247MB) going to heap size, and JMX running you are right at maxed out on ram, not to mention if anything else is running.

You might try lowering the Heap size to 2500MB, you may also want to consider a dedicated Wowza server if you are trying to use the machine for other activities.

Salvadore

9

Also, make sure you are utilizing all your CPU resources by tuning your VHost.xml file to use 4 cores, as shown in the tuning guide

Salvadore

10

Try making the .stream file on the edge that points to the origin like this:

{
    uri:"rtmp://[wowza-address]:1935/liveorigin/myStream",
    lsrResetOnStreamNotFound: false
}

Salvadore