out of memory with 64gb of ram...

Wowza Streaming Engine
1

Hello

i have a recurring memory issue illustrated with the following error log:

java.sql.SQLException: java.lang.OutOfMemoryError: Java heap space

i’ve implemented somes mysql connections in my custom module. running ubuntu 14, 64gb of ram. java heap size set to 10240MB

could it be related to my mysql functions ? would it be safe to increase the heap size, or 10240 is the maximum ?

also, this error usually occurs after this: (its a edge / origin config. the errors and displayed logs occurs on the origin server)

LB XMLParser: xmlParse() Timed out reading from 'http://xx.xx.xx.xx:1935/loadbalancerInterface?command=AAAAd5zxHRe9ucZsWWA2x7dhguH8Ox5eXEme%252BNHHPVNrBs0Rh%252Be7LsmBfQGTA%252Fb%252FpKU215RAhMog%250ArWNjfTgYzIrTIeVKEAgW%252FKpd%252BA0wBqO%252BSEHaE61I1%252BrQBakyjGYm7pX2YVrWf9dSZpQK0YCl675F%250AJVaOZV%252BNJrSyRQ%253D%253D': java.net.SocketTimeoutException: Read timed out|at java.net.SocketInputStream.socketRead0(Native Method)|at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)|at java.net.SocketInputStream.read(SocketInputStream.java:170)|at java.net.SocketInputStream.read(SocketInputStream.java:141)|at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)|

any clue about something i might have done wrong ?

i understand that the timeout could be netword related, but a bit suprising considering that both edge and origins are in the same datacenter

thank you for your advices

2

Hi,

We don’t recommend going beyond 10GB of Java heap space. I would certainly look at seeing if you can replicate this without any custom modules being involved in the mix.

You could also look at adding the following to [wowza-install-path]/conf/Tune.xml (within the tag)

<VMOption>-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath="${com.wowza.wms.AppHome}/logs"</VMOption>

It may provide some insight if it happens again. If it is replicable within the core Wowza product without any custom code enabled then I’d suggest raising this as a support ticket, quoting this forum thread. The heap dump file would also be very useful.

Paul

3

Hi,

If you check your access logs (found in your [install-dir]/logs, and you find SKIP messages, this normally means that your hardware/OS is struggling to keep up with your transcoding process. This would depend on the quality of the video that you are ingesting and decoding, and the quality of the output bit rate renditions that you are encoding. If you find these, you might want to tune your server using our Tuning guide.

Michelle

4

I was running into the same problems, with 32G of memory. The only custom module running is a slight modification of the one found here: http://thewowza.guru/application-http-provider-manager/, which as far as I can tell shouldn’t be doing anything for most of the time. Our setup at the moment is running about 14 live streams across 2 applications.

The problem for me seems to have started once I turned on the transcoder add-on (transrating audio on 1 stream, and 1 stream involving a full transcoding, though the latter stream is sometimes not connected). After running for 2-3 days (often over a weekend), Wowza throws the “Unexpected exception.: java.lang.OutOfMemoryError: Java heap space|” error. The real problem is that Wowza basically crashes, but gives no sign of it: all streams show up as connected and recording, even though they had not been working for about a day and a half as of this morning.