Understanding Garbage Collection options

I’ll just clear things up a bit:

  1. Server:
CPU: 2x Quad-core Intel Xeon
RAM: 4GB
OS: Linux 2.6.26-2-amd64 #1 SMP Fri Mar 27 04:02:59 UTC 2009 x86_64 GNU/Linux

Server has 2 Gbit Etherner interfaces - one is currently connected to ISP which guarantees 100Mbit bandwith for outgoing traffic.

  1. Java:
java version "1.6.0_12"
Java(TM) SE Runtime Environment (build 1.6.0_12-b04)
Java HotSpot(TM) 64-Bit Server VM (build 11.2-b01, mixed mode)

  1. Wowza Media Server Pro version 1.7.0_patch21 configured as shown in General tuning instructions and Linux and OSX tuning, running the “server” Java VM (tuning). Server is launched as standalone app with thease parameters:
-Xmx3000M -Djava.net.preferIPv4Stack=true -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:NewSize=1024m

or with -server parameter, but that performed worse.

  1. System serves two applications:

a) stream type “file” - serving mp3 files at 192kbit (up to 320kbit) - usualy around 60-80 concurrent connections

b) stream type “live” - serving H.264 video at 350kbit and AAC audio at 56kbit - usualy around 10-20 concurrent connections (one broadcast runs for up to 3 hours)

  1. Adobe Flash Media Live Encoder version 3 is used to encode live broadcasts and send to “live” application

The problem:

Garbage collector works fine with mp3 application and Heap never passes over the 1GB mark, but when live broadcast is introduced after 30-60 minutes of live streaming CMS old gen heap starts suddenly filling up and nothing seems to clean it up. After it reaches the maximum (3000MB) Wowza server becomes non-responsive and has to be killed and started-up again. Outgoing traffic usualy does not grow more than 50Mbit (info we got from the ISP).