Hi,
The general tuning suggest the following options for server JVM 1.6:
-XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:NewSize=1024m
Now what I am seeing with these values in JConsole is:
- Memory usage grows to about 1024m and then GC is invoked that results in about 1 second pause for the process
Somehow I don’t think this is the best option of a streaming server
Can anybody elaborate on these options, I tried reading “GC tuning” articles but it is all foreign to my area of expertise
Right now I am running the server with the following options
-Xms1000m -Xmx3000m -XX:MaxNewSize=16m -XX:NewSize=4m -XX:+CMSIncrementalMode -XX:+UseConcMarkSweepGC
which results in a lot of ParNew collections that only take very small fractions of a second, but I see no ConcurrentMarkSweep collections at all
Thanks!
Art
You make us sound like “the man”. Garbage collecting tuning has been tough. We are probably going to head into another round of tuning with Sun on their hardware sometime in October. I hope to have more precise recommendations come out of this effort. It certainly seems like a black art at this point. I have read the same GC documentation and have a hard time making heads or tails of it as well. The settings we are recommending are used by one of our large customers. They seems to have had a lot of luck. Again, they really aren’t doing low latency streaming.
I have read there is a new GC mechanism in Java 7. Sounds very promising. Seems like it addresses this exact problem:
http://tech.puredanger.com/java7/#g1
I think Java 7 is pretty stable at this point. So it might be worth a try.
Charlie
How many max concurrent connections? What is the bitrate of the streams? How many unique streams are there? What is the bitrate of your connection to the internet?
Charlie
I suspect that you might not really be able to get 100Mbps out of your connection. That would explain what you are seeing. Is there any way to test the connection speed that is available?
Charlie
Is everybody running Wowza with default settings and not questioning their usefulness or validity?
Any experts on java low pause garbage collection out here?
Art
I really don’t think it is a GC issue. We have folks running similar hardware that are pushing close to 1.5 Gbps out of two bonded 1GB nics with a little more memory without any issues. So it is most likely something else. It sounds like it could be a memory leak or something strange with the setup. Can you setup JConsole/JMX access to the server so I can watch it over the Internet. The instructions are in the User’s Guide (monitoring and management chapter). You can send the login information to support@wowza.com. I would like to watch the server the next time you do a live event. It would also help to take a heap dump of the server mid-event to see if we see any memory leak issues.
Are you running any custom server side modules?
Charlie
Zip up and send me the bin, logs and conf folders. It would also help to get a heap dump (these can be quite large). If you have the JDK running you can take a heap dump while the server is running with this command:
jmap -dump:format=b,file=snapshot2.jmap [PID_OF_PROCESS]
Where [PID_OF_PROCESS] is the process id of the Java process of the Wowza server. I suggest taking the heap dump as the memory starts to runnup and when the memory is full. The dumps are quite large so you will most likley need to post them somewhere for download.
You can send all this to support@wowza.com.
Charlie
I would appreciate any suggestions at this point, it’s beyond black art for me 
But I mean, there are a lot of people running Wowza, right? Somebody must have tweaked it and actually understood what they were doing
My posted settings don’t work too well either as they result in too many GCs (although very minor) and don’t translate too well when a broadcast is in progress… Tuning an idle server would be idiotic anyway, I have the load tester agreement somewhere, I think I’ll sign it and do my own load testing with numbers…
Art
Trying out jdk1.7.0 build 72, I’m not sure I want to use it in production, we’ll see
OKAY,
I took one for the team, running 1.7.0 b72 IN PRODUCTION
So far, looks good, barely noticeable stops in the stream (but still some).
No spikes in live delay, have been running default settings for over 6 hours with 8000 PS Scavenge collections taking overall about 12.5 seconds.
No idea what that means, but certainly java 1.7 show a lot of improvement in live low latency streaming performance
Hope this helps somebody
Art
Has anyone done some serious research in Garbage Collection for Wowza?
We’ve been struck by a big problem now. One of our client’s uses Wowza for live broadcasts - it’s running Linux 2.6.26-2-amd64 on 2 quad-core CPU server with 4GB of RAM and. We use Flash Media Live Encoder (version 3) to encode mixed down video and stream it to Wowza server (version 1.7.0_patch21) which has an application with stream type “live” to which all other viewers connect and watch the broadcast. After running the broadcast for about 30 minutes heap suddenly sky-rockets to the maximum and after few more minutes Wowza server stops responding.
Wowza server has been configured with settings shown in “General tuning instructions” and running with thease settings:
-Xmx3000M -Djava.net.preferIPv4Stack=true -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:NewSize=1024m
As you can see in this image taken from JConsole - there we started up Wowza server and after it reached heap limit we had to kill it and start it over again. The difference is that at first we started it up with thease settings:
-d64 -server -Xmx3000M -Djava.net.preferIPv4Stack=true -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:NewSize=1024m
But the -server mode was actualy performing worse.
So my question here is what could be the problem with GC? I also must add that this high peak is actualy “CMS old gen” - so as far as my understanding for GC goes - it’s those objects which live through “new” age, “medium” age and then finaly are considered “old”, but nobody goes up there to clean them out when they are un-referenced in “old gen” heap (or atleast somebody goes there when it’s already too late). Is there some specific java options to specify GC life-time for each of those generations?
Gusts
PS. I forgot to add that wowza stops responding and flooded error log full with thease:
- - - 2009-11-08 20:35:00 - - - - - 0.902 - - - - - - - -
java.lang.OutOfMemoryError: Java heap space
- - - 2009-11-08 20:35:00 - - - - - 0.902 - - - - - - - -
java.lang.OutOfMemoryError: Java heap space
at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:39)
at java.nio.ByteBuffer.allocate(ByteBuffer.java:312)
at org.apache.mina.common.SimpleByteBufferAllocator.allocate(Unknown Source)
at org.apache.mina.common.ByteBuffer.allocate(Unknown Source)
at org.apache.mina.transport.socket.nio.SocketIoProcessor.read(Unknown Source)
at org.apache.mina.transport.socket.nio.SocketIoProcessor.process(Unknown Source)
at org.apache.mina.transport.socket.nio.SocketIoProcessor.access$600(Unknown Source)
at org.apache.mina.transport.socket.nio.SocketIoProcessor$Worker.run(Unknown Source)
at org.apache.mina.util.NamePreservingRunnable.run(Unknown Source)
at java.lang.Thread.run(Thread.java:619)
Yes sorry I didn’t mention that. So first of all we also have one more application running which is stream type “file” and basicaly servers mp3 streaming. At that moment when wowza froze it was 17 open connections for live stream and 54 for mp3 application. Maximum was about 20 and 80 (so 100 concurent connections total). And it’s one stream per connection. Mp3 files are at 192kbps and the live stream was 386kbps (320kbps video, 56kbps audio). Server is connected to an ISP which offers 100Mbps bandwith.
Also server is running on:
java version “1.6.0_12”
Java™ SE Runtime Environment (build 1.6.0_12-b04)
Java HotSpot™ 64-Bit Server VM (build 11.2-b01, mixed mode)
Well from my calculations it should be able to push trough up to 250 live streams (according to bitrate) and for mp3 streams it’s even more because the bitrate is roughly two times smaller. Anyway on previous occasions we have had even more viewers (up to 100 for this live broadcast) and at that time server reported something around 50Mbit outgoing traffic.
I think I need to do one more test - involving 2-3 viewers and live broadcast for up to 2 hours - it seems for me the problem is in long length uninterrupted streams - in our case live music events.
I’ll just clear things up a bit:
- Server:
CPU: 2x Quad-core Intel Xeon
RAM: 4GB
OS: Linux 2.6.26-2-amd64 #1 SMP Fri Mar 27 04:02:59 UTC 2009 x86_64 GNU/Linux
Server has 2 Gbit Etherner interfaces - one is currently connected to ISP which guarantees 100Mbit bandwith for outgoing traffic.
- Java:
java version "1.6.0_12"
Java(TM) SE Runtime Environment (build 1.6.0_12-b04)
Java HotSpot(TM) 64-Bit Server VM (build 11.2-b01, mixed mode)
- Wowza Media Server Pro version 1.7.0_patch21 configured as shown in General tuning instructions and Linux and OSX tuning, running the “server” Java VM (tuning). Server is launched as standalone app with thease parameters:
-Xmx3000M -Djava.net.preferIPv4Stack=true -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:NewSize=1024m
or with -server parameter, but that performed worse.
- System serves two applications:
a) stream type “file” - serving mp3 files at 192kbit (up to 320kbit) - usualy around 60-80 concurrent connections
b) stream type “live” - serving H.264 video at 350kbit and AAC audio at 56kbit - usualy around 10-20 concurrent connections (one broadcast runs for up to 3 hours)
- Adobe Flash Media Live Encoder version 3 is used to encode live broadcasts and send to “live” application
The problem:
Garbage collector works fine with mp3 application and Heap never passes over the 1GB mark, but when live broadcast is introduced after 30-60 minutes of live streaming CMS old gen heap starts suddenly filling up and nothing seems to clean it up. After it reaches the maximum (3000MB) Wowza server becomes non-responsive and has to be killed and started-up again. Outgoing traffic usualy does not grow more than 50Mbit (info we got from the ISP).
Currently it is not quite possible - because we have to write inquiry to ISP and they will probably say something about security concerns. For now we have gathered some stats and Wowza has again got frozen:
Current heap size:*
2*963*014 kbytes
Maximum heap size:*
2*967*168 kbytes
Committed memory:*
2*967*168 kbytes
Pending finalization:*
0 objects
Garbage collector:*
Name = 'ParNew', Collections = 9*880, Total time spent = 25 minutes
Garbage collector:*
Name = 'ConcurrentMarkSweep', Collections = 2*017, Total time spent = 1 hour 4 minutes
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
28416 root 20 0 5105m 3.2g 4656 S 0 82.1 2418:13 /var/install/java/jre1.6.0_17/bin/java -Xmx3000M -Djava.net.preferIPv4Stack=true -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:NewSize=1024m -Dcom.sun.management.jmxremote=true -Dcom.wowza.wms.AppHome=/usr/local/WowzaMediaServerPro -Dcom.wowza.wms.ConfigURL= -Dcom.wowza.wms.ConfigHome=/usr/local/WowzaMediaServerPro -cp /usr/local/WowzaMediaServerPro/bin/wms-bootstrap.jar com.wowza.wms.bootstrap.Bootstrap start
VIRT = 5105m
RES = 3.2g
And the mysterious thing is that although server has 4GB of RAM and java is allowed to use up to 3GB it somehow has filled up the swap disk. And again its:
- - - 2009-11-09 15:58:15 - - - - - 1.519 - - - - - - - -
java.lang.OutOfMemoryError: Java heap space
ERROR server comment 2009-11-13 13:24:48 - - - - - 336393.634 - - - - - - - invoke(seek): java.lang.OutOfMemoryError: Java heap space: unknown
