java.lang.OutOfMemoryError error on 32GB system

Wowza Streaming Engine
1

Hi,

One of our customer use wowza server that has 32GB RAM and 24 core processor. They have 600 live channels on this machine - each channel consisting of 5 different bitrate streams (max 512 kbps). So in all 3000 streams published to wowza.

During popular events, they get around 50000 hits per hour. The CPU utilized is around 55%-60% at this load. However the wowza system crashes in some time and gives the outofmemory error as follows:

java.lang.OutOfMemoryError: unable to create new native thread

at java.lang.Thread.start0(Native Method)

at java.lang.Thread.start(Thread.java:657)

at org.apache.mina.util.NewThreadExecutor.execute(NewThreadExecutor.java:34)

at org.apache.mina.transport.socket.nio.SocketIoProcessor.startupWorker(SocketIoProcessor.java:104)

at org.apache.mina.transport.socket.nio.SocketIoProcessor.addNew(SocketIoProcessor.java:83)

at org.apache.mina.transport.socket.nio.SocketAcceptor$Worker.processSessions(SocketAcceptor.java:423)

at org.apache.mina.transport.socket.nio.SocketAcceptor$Worker.run(SocketAcceptor.java:331)

at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:43)

at java.lang.Thread.run(Thread.java:679)

They use latest wowza 3.6.3 version. Heap memory allocated to wowza server is 28 GB. All other options are default settings.

Are they publishing too many streams to wowza? What is the upper limit considering the above machine? Should the heap memory allocated be further increased? Or could the system be running out of threads? Number of open files limit (as shown by ulimit -a) is set at 32000.

Please help out urgently as there’s an event tomorrow.

Thanks

Nitin

2

What are your garbage collection settings on the system? As far as published streams, our general recommendation is no more than ~250 published streams. If you are doing hls you are likely losing a lot to the packetization process.

3

We are using the default setenv.sh file. So haven’t tinkered with GC settings. Do we have to change any of the GC settings? We dont know much about GC and we read in wowza tuning guide that its best to let the GC settings unchanged.

We are not using packetization. We have turned off all http packetizers.

4

Restart your Wowza instance and then zip up your [install-dir]/logs and [install-dir]/conf directories and send it to support@wowza.com. Include a reference link to this post.