Wowza server hanged "java.lang.OutOfMemoryError: Java heap space "

Wowza Streaming Engine
1

Hi,

We have setup wowza in origin-edge configuration. Earlier it was streaming 3 channels from origin and it was operating fine. Then we installed NVIDIA K4000 Card and we setup 8 channels at the origin. Everything was going fine as expected. But suddenly, yesterday, the origin wowza server just got hanged. By hanged, I mean the service was running, but it was not processing the stream.

I checked the log there was message like:

WARN server comment 2013-09-15 07:07:02 - - - - - 235821.696 - - - - - - - - Unexpected exception.

java.lang.OutOfMemoryError: Java heap space

at java.nio.HeapByteBuffer.(HeapByteBuffer.java:39)

at java.nio.ByteBuffer.allocate(ByteBuffer.java:312)

at org.apache.mina.common.SimpleByteBufferAllocator.allocate(Unknown Source)

at org.apache.mina.common.ByteBuffer.allocate(Unknown Source)

at org.apache.mina.transport.socket.nio.SocketIoProcessor.read(Unknown Source)

at org.apache.mina.transport.socket.nio.SocketIoProcessor.process(Unknown Source)

at org.apache.mina.transport.socket.nio.SocketIoProcessor.access$600(Unknown Source)

at org.apache.mina.transport.socket.nio.SocketIoProcessor$Worker.run(Unknown Source)

at org.apache.mina.util.NamePreservingRunnable.run(Unknown Source)

at java.lang.Thread.run(Thread.java:662)

WARN server comment 2013-09-15 07:07:05 - - - - - 235841.725 - - - - - - - - Unexpected exception.

java.lang.OutOfMemoryError: Java heap space

at java.nio.HeapByteBuffer.(HeapByteBuffer.java:39)

at java.nio.ByteBuffer.allocate(ByteBuffer.java:312)

at org.apache.mina.common.SimpleByteBufferAllocator.allocate(Unknown Source)

at org.apache.mina.common.ByteBuffer.allocate(Unknown Source)

at org.apache.mina.transport.socket.nio.SocketIoProcessor.read(Unknown Source)

at org.apache.mina.transport.socket.nio.SocketIoProcessor.process(Unknown Source)

at org.apache.mina.transport.socket.nio.SocketIoProcessor.access$600(Unknown Source)

at org.apache.mina.transport.socket.nio.SocketIoProcessor$Worker.run(Unknown Source)

at org.apache.mina.util.NamePreservingRunnable.run(Unknown Source)

at java.lang.Thread.run(Thread.java:662)

WARN server comment 2013-09-15 07:07:05 - - - - - 235829.413 - - - - - - - - Unexpected exception.

I checked the forum and found this issue is similar to the issue mentioned at;

https://www.wowza.com/forums/showthread.php?5504-java-lang-OutOfMemoryError-Java-heap-space

But, didn’t find any solution in that thread.

I have already tuned my Wowza as per :

https://www.wowza.com/docs/general-tuning

Any solution of this?

Thanks,

Birendra

2

Hello Birendra,

You should try to grab a heap dump the next time you receive this exception and send it to support@wowza.com along with a zipped file of your conf & logs.

Matt

3

I can report a similar OOM problem.

Running Software: Wowza Media Server 3.6.3 build8031

java version “1.7.0_25”

OpenJDK Runtime Environment (IcedTea 2.3.10) (7u25-2.3.10-1~deb7u1)

OpenJDK 64-Bit Server VM (build 23.7-b01, mixed mode)

On a Linux Debian Wheezy server, Intel 1270 CPU , 16 Gb of memory.

Wowza has 8 Gb (XmX) allocated. Very low load (about 2%). The server has a server-sided playlist , running an 700 Mb , 12 minute, +/- 5 Mbit MP4 file for 24/7

An external process (ffmpeg) reads that (RTMP) stream and feeds it back into Wowza (after transcoding).

At startup the Wowza process uses around 630 Mb (RES). At this moment, about 6 hours after starting, the process is using 702 Mb (RES). This is normal and not causing the final OOM ; if Wowza would “leak” 10 Mb per hour then even after 48 hours the total leakage would not be more than 500 Mb (which is very acceptable to us).

However at some point, in 1 or 2 days suddenly memory usage will go up a lot ( to 8000 Mb ) and there will be OOM errors in the log.

This origin also feeds 2 edges, but the load is low (about 2-5 concurrent viewers maximum).

This is a test-environment but we can’t get it stable. The Edges are stable. The edges are located at about 40ms and 190ms distance.

I’ve prepared the options to dump the heap, when that has happened we’ll send it to Wowza if they want to have a look.

4

The problem was ‘solved’ by setting the StreamType of the Origin from repeater to “live”. According to the feedback from Wowza the origin was buffering video for the edge and due to a possible underperforming network it would keep buffering all the way to the maximum available memory and then crash. Now the Origin is ‘stable’ at around 900 Mb of memory usage after 4 days of operation under the same load.

5

The best I can do here is suggest you follow matt_y’s advice as stated above:

“You should try to grab a heap dump the next time you receive this exception and send it to support@wowza.com along with a zipped file of your conf & logs.”

Here is a guide for taking a heap dump:

How to take a Java heap dump

Salvadore

6

We are having the same issue running Wowza on Amazon EC2

java.lang.OutOfMemoryError: Java heap space

appears in the logs repeatedly, CPU goes to 100%, wowza does not stream and becomes unresponsive.

we also get this stack trace in the error log

java.lang.OutOfMemoryError: Java heap space

at java.nio.HeapByteBuffer.(HeapByteBuffer.java:57)

at java.nio.ByteBuffer.allocate(ByteBuffer.java:331)

at org.apache.mina.common.SimpleByteBufferAllocator.allocate(Unknown Source)

at org.apache.mina.common.ByteBuffer.allocate(Unknown Source)

at org.apache.mina.transport.socket.nio.SocketIoProcessor.read(Unknown Source)

at org.apache.mina.transport.socket.nio.SocketIoProcessor.process(Unknown Source)

at org.apache.mina.transport.socket.nio.SocketIoProcessor.access$600(Unknown Source)

at org.apache.mina.transport.socket.nio.SocketIoProcessor$Worker.run(Unknown Source)

at org.apache.mina.util.NamePreservingRunnable.run(Unknown Source)

at java.lang.Thread.run(Thread.java:724)

The Amazon AMI ID is ami-4669142f.

We have not done any special tuning (no adjusting OS, Java, or Wowza memory or other limits), on the assumption that the pre-built AMI would be correctly configured “out-of-the-box”.

We have enabled RTMP PushPublishing according to the instructions available on the Internet and in general it seems to be working as designed.

The OutOfMemory errors are coming under a light server load, for example 1 stream with 3 bitrates, 700k, 400k, 200k, and an occasional VOD from S3.

This has happened twice so far.

7

Usually, this error is thrown when the Java Virtual Machine cannot allocate an object because it is out of memory, and no more memory could be made available by the garbage collector.

Therefore you pretty much have two options:

  • Increase the default memory your program is allowed to use using the -Xmx option (for instance for 1024 MB: -Xmx1024m)

  • Modify your program so that it needs less memory, using less big data structures and getting rid of objects that are not any more used at some point in your program

Increasing the heap size is a bad solution, 100% temporary. It will crash again in somewhere else. To avoid these issues, write high performance code.

  • Use local variables wherever possible.
  • Make sure you select the correct object (EX: Selection between String, StringBuffer and StringBuilder)
  • Use a good code system for your program(EX: Using static variables VS non static variables)
  • Other stuff which could work on your code.
  • Try to move with Multy Threading
1 Like