Tuning values 16 gb ram

[Vasya_Volkov]

Hi,

Our server has 16 Gb ram. We use linux, latest wowza and latest Oracle’s jdk.

Here is head of our setenv.sh

#!/bin/sh
_EXECJAVA=java
JAVA_OPTS="$JAVA_OPTS -Xmx10000M"
# Uncomment to run server environment (faster), Note: will only work if server VM install, comes with JDL
JAVA_OPTS="$JAVA_OPTS -server"
# Can be a better GC setting to avoid long pauses
JAVA_OPTS="$JAVA_OPTS -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:+UseLargePages -XX:+UseParNewGC -XX:NewSize=512m"

And wowza works fine about 10 hours. But than memory which it allocates grows from 11Gb to 50Gb.

We have 50 1 Mbit channels 50 50Kbit and 50 200Kbit input channels (to achive adaptive bitrate) with nDVR which writes only 1 Mbit channels.

How can we tune this?

We also can drop caches like: /bin/sync; echo 1 > /proc/sys/vm/drop_caches

Thanks.

[Randall_Auriemma]

If you are running any custom modules you can investigate using JConsole as mentioned in the User Guide.

[Jason_Hilton]

Hi

Take a look at the tuning guide here,

https://www.wowza.com/docs/how-to-do-performance-tuning

Jason

[Richard_Lanham]

We don’t recommend setting java heap size that high. Try 6000m - 8000M

JAVA_OPTS="$JAVA_OPTS -Xmx6000M"

And try without the last GC setting. But you might make the java heap size change first, separately. See how much that improves, then try without the GC setting for long pauses.

Richard

[Richard_Lanham]

The JConsole screen shots look normal. If you have no custom modules and you are hitting outOfMemory it is most likely simply that are overloading the server.

We have 50 1 Mbit channels 50 50Kbit and 50 200Kbit input channels (to achive adaptive bitrate) with nDVR which writes only 1 Mbit channels.

That’s a lot of incoming streams. Are you using the Wowza Transcoder? Either way, it’s a lot. I think you will have to add hardware.

Richard

[Richard_Lanham]

The snapshot looks normal anyway, but obviously you are heading to this outOfMemory so something is not okay. From my understanding, and experience with other support cases, the number of incoming streams has its own limits, bandwidth is not the limiting factor, as with number of subscribing clients, it is the server. 100 incoming streams is very good, that is about the usual limit for a typical high performance server. The best I have heard of is 200. You have 150. I would split that between two good (late model quad or dual quad cpu, 4 - 8g ram, 64bit OS/Java, 1 - 4gbs nic) servers.

Richard

[Andrew_Kennedy]

Have a look at this thread

Performance issues on a local LAN

We’re trying to track down a performance problem and isolate where it would be coming from. Here is the scenario: Wowza is a 4 core Windows box and has 8GB of RAM. We have followed all the steps in the performance tuning guide No firewalls involved for this test as it is all LAN to LAN on the same subnet Wowza Java heap size is set to 5000 Wowza has the -server option enabled Java JDE version is 1.7.0_01 We do not have garbage collection enabled The Wowza application I’m …

which covers some detailed GC settings (last post) which may help.

Shamrock

[Vasya_Volkov]

Hi,

No I don’t use any custom modules except that your team provide.

Richard what should I investigate with Jconsole? If I set small heap size like 8gb and G1experimental settings to GC my wowza just hangs after some time with OutOfMemory exception. And what can I do? If I try Incremental mode with 10gb wowza allocates 40gb and hangs after some time because we have no memory.

There are some data when things became bad and wowza hangs:

ps aux
USER****** PID %CPU %MEM*** VSZ** RSS TTY***** STAT START** TIME COMMAND
root***** 3907* 314* 7.4 34348532 1220848 pts/2 Sl+ Mar25 3635:06 /usr/lib/jvm/oracle-jdk-bin-1.7/bin/java
VSZ=34348532 = 32Gbyte
RSS=1220848 = 1.2Gbyte
# cat meminfo
MemTotal:****** 16405628 kB
MemFree:******** 2974680 kB
Buffers:************ 300 kB
Cached:********** 130044 kB
SwapCached:*********** 0 kB
Active:********* 2204512 kB
Inactive:******** 119440 kB
Active(anon):*** 2193952 kB
Inactive(anon):**** 1300 kB
Active(file):***** 10560 kB
Inactive(file):** 118140 kB
Unevictable:********** 0 kB
Mlocked:************** 0 kB
SwapTotal:************ 0 kB
SwapFree:************* 0 kB
Dirty:************** 188 kB
Writeback:************ 0 kB
AnonPages:****** 2193972 kB
Mapped:*********** 11240 kB
Shmem:************* 1300 kB
Slab:************ 430208 kB
SReclaimable:**** 228308 kB
SUnreclaim:****** 201900 kB
KernelStack:***** 258760 kB
PageTables:******* 68532 kB
NFS_Unstable:********* 0 kB
Bounce:*************** 0 kB
WritebackTmp:********* 0 kB
CommitLimit:**** 3123772 kB
Committed_AS:** 33835988 kB
VmallocTotal:** 34359738367 kB
VmallocUsed:****** 77508 kB
VmallocChunk:** 34359656532 kB
HardwareCorrupted:**** 0 kB
AnonHugePages:***** 2048 kB
HugePages_Total:*** 4960
HugePages_Free:****** 25

[Vasya_Volkov]

Hi,

Recently we set heap size 9 Gb and here you can see what we get:

Is it normal that live threads value grows?

At now we use heap size 6Gb but also get the same:

Number of thread grows…

Do you need more information like dump thread stack or dump heap size?

What’s wrong?

[Vasya_Volkov]

We have got our own transcoder which has good quality of compression.

What’s the problem? Why this number of channels so lot?

You say that our graphs is normal? So it is normal that live threads grows up?

With default wowza settings our wowza hangs after 7 hours.

top - 18:00:44 up  5:48,  4 users,  load average: 0.94, 1.00, 1.15
Tasks: 123 total,   1 running, 122 sleeping,   0 stopped,   0 zombie
Cpu(s):  8.4%us,  0.2%sy,  0.0%ni, 91.4%id,  0.0%wa,  0.0%hi,  0.1%si,  0.0%st
Mem:  16405628k total, 16160040k used,   245588k free,    12476k buffers
Swap:        0k total,        0k used,        0k free,  8237652k cached

INFO server comment - LiveStreamPacketizerCupertino.init[live/_definst_/vopr_i_otv_576p.stream]: chunkDurationTarget: 3000 - -
INFO server comment - LiveStreamPacketizerCupertino.init[live/_definst_/vopr_i_otv_576p.stream]: chunkDurationTolerance: 500 - -
INFO server comment - LiveStreamPacketizerCupertino.init[live/_definst_/vopr_i_otv_576p.stream]: audioGroupCount: 3 - -
Exception in thread "Thread-2415" Exception in thread "VHostTransport._defaultVHost_.172" Exception in thread "VHostTransport._defaultVHost_.127" Exception in thread "Thread-149" Exception in thread "Thread-6927" Exception in thread "Thread-911" Exception in thread "Thread-76" Exception in thread "VHostHandler._defaultVHost_.242" Exception in thread "VHostHandler._defaultVHost_.398" Exception in thread "Thread-334"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Thread-83"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Thread-252"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Thread-931"
Exception in thread "VHostHandler._defaultVHost_.144" Exception in thread "Thread-141"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Thread-3841"
Exception in thread "Thread-90"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Thread-3843"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Thread-300"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Thread-127"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Thread-190"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "VHostTransport._defaultVHost_.255"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Thread-130"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Thread-355"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Thread-278"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "VHostTransport._defaultVHost_.268"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "VHostTransport._defaultVHost_.240"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Thread-115"

[Vasya_Volkov]

I think we can reduce heap size because of we don’t catch error like “Java heap size”.

Our errors from exceptions in threads which wowza generates many and after it jvm didn’t reduce this value, it’s only grows. Ho we can influence on this?

How much memory “common” thread of such scenario needs? I mean how small can I set Xss parameter?

I post here our limits in system:

ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 128146
max locked memory       (kbytes, -l) 12582912
max memory size         (kbytes, -m) unlimited
open files                      (-n) 32768
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 32000
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

thread-max:

cat /proc/sys/kernel/threads-max
256292

java:

java -version
java version "1.7.0_03"
Java(TM) SE Runtime Environment (build 1.7.0_03-b04)
Java HotSpot(TM) 64-Bit Server VM (build 22.1-b02, mixed mode)

Here is our server info:

Kernel:

Linux 3.2.9-hardened-r1 #5 SMP Tue Mar 27 12:04:41 MSK 2012 x86_64 Intel(R) Xeon(R) CPU E5649 @ 2.53GHz GenuineIntel GNU/Linux

CPU:

cat /proc/cpuinfo 
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 44
model name	: Intel(R) Xeon(R) CPU           E5649  @ 2.53GHz
stepping	: 2
microcode	: 0x14
cpu MHz		: 2533.625
cache size	: 12288 KB
physical id	: 0
siblings	: 12
core id		: 0
cpu cores	: 6
apicid		: 0
initial apicid	: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 popcnt aes lahf_lm ida arat epb dts tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5067.25
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:
processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 44
model name	: Intel(R) Xeon(R) CPU           E5649  @ 2.53GHz
stepping	: 2
microcode	: 0x14
cpu MHz		: 2533.625
cache size	: 12288 KB
physical id	: 0
siblings	: 12
core id		: 8
cpu cores	: 6
apicid		: 16
initial apicid	: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 popcnt aes lahf_lm ida arat epb dts tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5066.81
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:
processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 44
model name	: Intel(R) Xeon(R) CPU           E5649  @ 2.53GHz
stepping	: 2
microcode	: 0x14
cpu MHz		: 2533.625
cache size	: 12288 KB
physical id	: 0
siblings	: 12
core id		: 2
cpu cores	: 6
apicid		: 4
initial apicid	: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 popcnt aes lahf_lm ida arat epb dts tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5066.79
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:
processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 44
model name	: Intel(R) Xeon(R) CPU           E5649  @ 2.53GHz
stepping	: 2
microcode	: 0x14
cpu MHz		: 2533.625
cache size	: 12288 KB
physical id	: 0
siblings	: 12
core id		: 10
cpu cores	: 6
apicid		: 20
initial apicid	: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 popcnt aes lahf_lm ida arat epb dts tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5066.79
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:
processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 44
model name	: Intel(R) Xeon(R) CPU           E5649  @ 2.53GHz
stepping	: 2
microcode	: 0x14
cpu MHz		: 2533.625
cache size	: 12288 KB
physical id	: 0
siblings	: 12
core id		: 1
cpu cores	: 6
apicid		: 2
initial apicid	: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 popcnt aes lahf_lm ida arat epb dts tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5066.79
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:
processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 44
model name	: Intel(R) Xeon(R) CPU           E5649  @ 2.53GHz
stepping	: 2
microcode	: 0x14
cpu MHz		: 2533.625
cache size	: 12288 KB
physical id	: 0
siblings	: 12
core id		: 9
cpu cores	: 6
apicid		: 18
initial apicid	: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 popcnt aes lahf_lm ida arat epb dts tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5066.81
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:
processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 44
model name	: Intel(R) Xeon(R) CPU           E5649  @ 2.53GHz
stepping	: 2
microcode	: 0x14
cpu MHz		: 2533.625
cache size	: 12288 KB
physical id	: 0
siblings	: 12
core id		: 0
cpu cores	: 6
apicid		: 1
initial apicid	: 1
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 popcnt aes lahf_lm ida arat epb dts tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5066.79
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:
processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 44
model name	: Intel(R) Xeon(R) CPU           E5649  @ 2.53GHz
stepping	: 2
microcode	: 0x14
cpu MHz		: 2533.625
cache size	: 12288 KB
physical id	: 0
siblings	: 12
core id		: 8
cpu cores	: 6
apicid		: 17
initial apicid	: 17
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 popcnt aes lahf_lm ida arat epb dts tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5066.80
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:
processor	: 8
vendor_id	: GenuineIntel
cpu family	: 6
model		: 44
model name	: Intel(R) Xeon(R) CPU           E5649  @ 2.53GHz
stepping	: 2
microcode	: 0x14
cpu MHz		: 2533.625
cache size	: 12288 KB
physical id	: 0
siblings	: 12
core id		: 2
cpu cores	: 6
apicid		: 5
initial apicid	: 5
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 popcnt aes lahf_lm ida arat epb dts tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5066.79
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:
processor	: 9
vendor_id	: GenuineIntel
cpu family	: 6
model		: 44
model name	: Intel(R) Xeon(R) CPU           E5649  @ 2.53GHz
stepping	: 2
microcode	: 0x14
cpu MHz		: 2533.625
cache size	: 12288 KB
physical id	: 0
siblings	: 12
core id		: 10
cpu cores	: 6
apicid		: 21
initial apicid	: 21
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 popcnt aes lahf_lm ida arat epb dts tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5066.81
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:
processor	: 10
vendor_id	: GenuineIntel
cpu family	: 6
model		: 44
model name	: Intel(R) Xeon(R) CPU           E5649  @ 2.53GHz
stepping	: 2
microcode	: 0x14
cpu MHz		: 2533.625
cache size	: 12288 KB
physical id	: 0
siblings	: 12
core id		: 1
cpu cores	: 6
apicid		: 3
initial apicid	: 3
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 popcnt aes lahf_lm ida arat epb dts tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5066.80
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:
processor	: 11
vendor_id	: GenuineIntel
cpu family	: 6
model		: 44
model name	: Intel(R) Xeon(R) CPU           E5649  @ 2.53GHz
stepping	: 2
microcode	: 0x14
cpu MHz		: 2533.625
cache size	: 12288 KB
physical id	: 0
siblings	: 12
core id		: 9
cpu cores	: 6
apicid		: 19
initial apicid	: 19
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 popcnt aes lahf_lm ida arat epb dts tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5066.79
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

And also our Server.xml and Vhost.xml

Server.xml:

<Root>
        <Server>
                <CommandInterface>
                        <HostPort>
                                <IpAddress>*</IpAddress>
                                <Port>8083</Port>
                        </HostPort>
                </CommandInterface>
                <AdminInterface>
                        <!-- Objects exposed through JMX interface: Server, VHost, VHostItem, Application, ApplicationInstance, MediaCaster, Module, Client, MediaStream, SharedObject, Acceptor, IdleWorker -->
                        <ObjectList>Server,VHost,VHostItem,Application,ApplicationInstance,MediaCaster,Client,MediaStream,Module,IdleWorker</ObjectList>
                </AdminInterface>
                <!-- JMXUrl: service:jmx:rmi://localhost:8084/jndi/rmi://localhost:8085/jmxrmi -->
                <JMXRemoteConfiguration>
                        <Enable>true</Enable>
                        <IpAddress>localhost</IpAddress> <!-- set to localhost or internal ip address if behind NAT -->
                        <RMIServerHostName>localhost</RMIServerHostName> <!-- set to external ip address or domain name if behind NAT -->
                        <RMIConnectionPort>8084</RMIConnectionPort>
                        <RMIRegistryPort>8085</RMIRegistryPort>
                        <Authenticate>false</Authenticate>
                        <PasswordFile>${com.wowza.wms.ConfigHome}/conf/jmxremote.password</PasswordFile>
                        <AccessFile>${com.wowza.wms.ConfigHome}/conf/jmxremote.access</AccessFile>
                        <SSLSecure>false</SSLSecure>
                </JMXRemoteConfiguration>
                <UserAgents>Shockwave Flash|CFNetwork|MacNetwork/1.0 (Macintosh)</UserAgents>
                <ServerListeners>
                        <!--
                        <ServerListener>
                                <BaseClass>com.wowza.wms.plugin.loadbalancer.ServerListenerLoadBalancerListener</BaseClass>
                        </ServerListener>
                        -->
                        <!--
                        <ServerListener>
                                <BaseClass>com.wowza.wms.plugin.loadbalancer.ServerListenerLoadBalancerSender</BaseClass>
                        </ServerListener>
                        -->
                </ServerListeners>
                <VHostListeners>
                        <!--
                        <VHostListener>
                                <BaseClass></BaseClass>
                        </VHostListener>
                        -->
                </VHostListeners>
                <HandlerThreadPool>
                        <PoolSize>10</PoolSize>
                </HandlerThreadPool>
                <TransportThreadPool>
                        <PoolSize>10</PoolSize>
                </TransportThreadPool>
                <RTP>
                        <DatagramStartingPort>6970</DatagramStartingPort>
                        <DatagramPortSharing>false</DatagramPortSharing>
                </RTP>
                <!-- Properties defined here will be added to the IServer.getProperties() collection -->
                <Properties>
                </Properties>
        </Server>
</Root>

VHost.xml:

	<VHost>
		<HostPortList>
			<HostPort>
				<ProcessorCount>12</ProcessorCount>
				<IpAddress>*</IpAddress>
				<!-- Separate multiple ports with commas -->
				<!-- 80: HTTP, RTMPT -->
				<!-- 554: RTSP -->
				<Port>1935,80</Port>
				<SocketConfiguration>
					<ReuseAddress>true</ReuseAddress>
					<!-- suggested settings for video on demand applications -->
					<ReceiveBufferSize>24000</ReceiveBufferSize>
					<SendBufferSize>65000</SendBufferSize>
					<!-- suggest settings for low latency chat and video recording applications
					<ReceiveBufferSize>16000</ReceiveBufferSize>
					<SendBufferSize>16000</SendBufferSize>
					-->
					<KeepAlive>true</KeepAlive>
					<!-- <TrafficClass>0</TrafficClass> -->
					<!-- <OobInline>false</OobInline> -->
					<!-- <SoLingerTime>-1</SoLingerTime> -->
					<!-- <TcpNoDelay>false</TcpNoDelay> -->
					<AcceptorBackLog>100</AcceptorBackLog>
				</SocketConfiguration>
				<HTTPStreamerAdapterIDs>cupertinostreaming,smoothstreaming,sanjosestreaming,dvrchunkstreaming</HTTPStreamerAdapterIDs>
				<HTTPProviders>
					<HTTPProvider>
						<BaseClass>com.wowza.wms.http.HTTPCrossdomain</BaseClass>
						<RequestFilters>*crossdomain.xml</RequestFilters>
						<AuthenticationMethod>none</AuthenticationMethod>
					</HTTPProvider>
					<HTTPProvider>
						<BaseClass>com.wowza.wms.http.HTTPClientAccessPolicy</BaseClass>
						<RequestFilters>*clientaccesspolicy.xml</RequestFilters>
						<AuthenticationMethod>none</AuthenticationMethod>
					</HTTPProvider>
					<HTTPProvider>
						<BaseClass>com.wowza.wms.http.HTTPProviderMediaList</BaseClass>
						<RequestFilters>*jwplayer.rss|*medialist.smil|*manifest-rtmp.f4m</RequestFilters>
						<AuthenticationMethod>none</AuthenticationMethod>
					</HTTPProvider>
					<HTTPProvider>
						<BaseClass>com.wowza.wms.http.HTTPServerVersion</BaseClass>
						<RequestFilters>*</RequestFilters>
						<AuthenticationMethod>none</AuthenticationMethod>
					</HTTPProvider>
				</HTTPProviders>
			</HostPort>
			
			<!-- 443 with SSL -->
			<!--
			<HostPort>
				<ProcessorCount>4</ProcessorCount>
				<IpAddress>*</IpAddress>
				<Port>443</Port>
				<SSLConfig>
					<KeyStorePath>${com.wowza.wms.context.VHostConfigHome}/conf/keystore.cert</KeyStorePath>
					<KeyStorePassword>[password]</KeyStorePassword>
					<KeyStoreType>JKS</KeyStoreType>
					<SSLProtocol>TLS</SSLProtocol>
					<Algorithm>SunX509</Algorithm>
					<CipherSuites></CipherSuites>
					<Protocols></Protocols>
				</SSLConfig>
				<SocketConfiguration>
					<ReuseAddress>true</ReuseAddress>
					<ReceiveBufferSize>24000</ReceiveBufferSize>
					<SendBufferSize>65000</SendBufferSize>
					<KeepAlive>true</KeepAlive>
					<AcceptorBackLog>100</AcceptorBackLog>
				</SocketConfiguration>
				<HTTPStreamerAdapterIDs>cupertinostreaming,smoothstreaming,sanjosestreaming,dvrchunkstreaming</HTTPStreamerAdapterIDs>
				<HTTPProviders>
					<HTTPProvider>
						<BaseClass>com.wowza.wms.http.HTTPCrossdomain</BaseClass>
						<RequestFilters>*crossdomain.xml</RequestFilters>
						<AuthenticationMethod>none</AuthenticationMethod>
					</HTTPProvider>
					<HTTPProvider>
						<BaseClass>com.wowza.wms.http.HTTPClientAccessPolicy</BaseClass>
						<RequestFilters>*clientaccesspolicy.xml</RequestFilters>
						<AuthenticationMethod>none</AuthenticationMethod>
					</HTTPProvider>
					<HTTPProvider>
						<BaseClass>com.wowza.wms.http.HTTPProviderMediaList</BaseClass>
						<RequestFilters>*jwplayer.rss|*medialist.smil|*manifest-rtmp.f4m</RequestFilters>
						<AuthenticationMethod>none</AuthenticationMethod>
					</HTTPProvider>
					<HTTPProvider>
						<BaseClass>com.wowza.wms.http.HTTPServerVersion</BaseClass>
						<RequestFilters>*</RequestFilters>
						<AuthenticationMethod>none</AuthenticationMethod>
					</HTTPProvider>
				</HTTPProviders>
			</HostPort>
			-->
			
			<!-- Admin HostPort -->
			<HostPort>
				<ProcessorCount>1</ProcessorCount>
				<IpAddress>*</IpAddress>
				<Port>8086</Port>
				<SocketConfiguration>
					<ReuseAddress>true</ReuseAddress>
					<ReceiveBufferSize>16000</ReceiveBufferSize>
					<SendBufferSize>16000</SendBufferSize>
					<KeepAlive>true</KeepAlive>
					<AcceptorBackLog>100</AcceptorBackLog>
				</SocketConfiguration>
				<HTTPStreamerAdapterIDs></HTTPStreamerAdapterIDs>
				<HTTPProviders>
					<HTTPProvider>
						<BaseClass>com.wowza.wms.http.streammanager.HTTPStreamManager</BaseClass>
						<RequestFilters>streammanager*</RequestFilters>
						<AuthenticationMethod>admin-digest</AuthenticationMethod>
					</HTTPProvider>
					<HTTPProvider>
						<BaseClass>com.wowza.wms.http.HTTPServerInfoXML</BaseClass>
						<RequestFilters>serverinfo*</RequestFilters>
						<AuthenticationMethod>admin-digest</AuthenticationMethod>
					</HTTPProvider>
					<HTTPProvider>
						<BaseClass>com.wowza.wms.http.HTTPConnectionInfo</BaseClass>
						<RequestFilters>connectioninfo*</RequestFilters>
						<AuthenticationMethod>admin-digest</AuthenticationMethod>
					</HTTPProvider>
					<HTTPProvider>
						<BaseClass>com.wowza.wms.http.HTTPConnectionCountsXML</BaseClass>
						<RequestFilters>connectioncounts*</RequestFilters>
						<AuthenticationMethod>admin-digest</AuthenticationMethod>
					</HTTPProvider>
					<HTTPProvider>
						<BaseClass>com.wowza.wms.transcoder.httpprovider.HTTPTranscoderThumbnail</BaseClass>
						<RequestFilters>transcoderthumbnail*</RequestFilters>
						<AuthenticationMethod>admin-digest</AuthenticationMethod>
					</HTTPProvider>
					<HTTPProvider>
						<BaseClass>com.wowza.wms.http.HTTPProviderMediaList</BaseClass>
						<RequestFilters>medialist*</RequestFilters>
						<AuthenticationMethod>admin-digest</AuthenticationMethod>
					</HTTPProvider>
					<HTTPProvider>
						<BaseClass>com.wowza.wms.plugin.dvrstreamrecord.HTTPDvrStreamRecord</BaseClass>
						<RequestFilters>dvrstreamrecord*</RequestFilters>
						<AuthenticationMethod>none</AuthenticationMethod>
					</HTTPProvider>
					<HTTPProvider>
						<BaseClass>com.wowza.wms.plugin.livestreamrecord.HTTPLiveStreamRecord</BaseClass>
						<RequestFilters>livestreamrecord*</RequestFilters>
						<AuthenticationMethod>none</AuthenticationMethod>
					</HTTPProvider>
					<HTTPProvider>
						<BaseClass>com.wowza.wms.http.HTTPServerVersion</BaseClass>
						<RequestFilters>*</RequestFilters>
						<AuthenticationMethod>none</AuthenticationMethod>
					</HTTPProvider>
				</HTTPProviders>
			</HostPort>
		
		</HostPortList>
		
		<HTTPStreamerAdapters>
			<HTTPStreamerAdapter>
				<ID>smoothstreaming</ID>
				<Name>smoothstreaming</Name>
				<Properties>
				</Properties>
			</HTTPStreamerAdapter>
			<HTTPStreamerAdapter>
				<ID>cupertinostreaming</ID>
				<Name>cupertinostreaming</Name>
				<Properties>
				</Properties>
			</HTTPStreamerAdapter>
			<HTTPStreamerAdapter>
				<ID>sanjosestreaming</ID>
				<Name>sanjosestreaming</Name>
				<Properties>
				</Properties>
			</HTTPStreamerAdapter>
			<HTTPStreamerAdapter>
				<ID>dvrchunkstreaming</ID>
				<Name>dvrchunkstreaming</Name>
				<Properties>
				</Properties>
			</HTTPStreamerAdapter>
		</HTTPStreamerAdapters>
		<HandlerThreadPool>
			<PoolSize>480</PoolSize>
		</HandlerThreadPool>
		<TransportThreadPool>
			<PoolSize>320</PoolSize>
		</TransportThreadPool>
		<IdleWorkers>
			<WorkerCount>12</WorkerCount>
			<CheckFrequency>50</CheckFrequency>
			<MinimumWaitTime>5</MinimumWaitTime>
		</IdleWorkers>
		<NetConnections>
			<ProcessorCount>12</ProcessorCount>
			<IdleFrequency>250</IdleFrequency>
			<SocketConfiguration>
				<ReuseAddress>true</ReuseAddress>
				<ReceiveBufferSize>65000</ReceiveBufferSize>
				<SendBufferSize>24000</SendBufferSize>
				<KeepAlive>true</KeepAlive>
				<!-- <TrafficClass>0</TrafficClass> -->
				<!-- <OobInline>false</OobInline> -->
				<!-- <SoLingerTime>-1</SoLingerTime> -->
				<!-- <TcpNoDelay>false</TcpNoDelay> -->
				<AcceptorBackLog>100</AcceptorBackLog>
			</SocketConfiguration>
		</NetConnections>
		<HTTPTunnel>
			<KeepAliveTimeout>2000</KeepAliveTimeout>
		</HTTPTunnel>
		<Client>
			<ClientTimeout>90000</ClientTimeout>
			<IdleFrequency>250</IdleFrequency>
		</Client>
		<!-- RTP/Authentication/Methods defined in Authentication.xml. Default setup includes; none, basic, digest -->
		<RTP>
			<IdleFrequency>75</IdleFrequency>
			<DatagramConfiguration>
				<Incoming>
					<ReuseAddress>true</ReuseAddress>
					<ReceiveBufferSize>2048000</ReceiveBufferSize>
					<SendBufferSize>65000</SendBufferSize>
					<MulticastBindToAddress>true</MulticastBindToAddress>
					<MulticastInterfaceAddress>172.30.33.29</MulticastInterfaceAddress>
					<!-- <TrafficClass>0</TrafficClass> -->
					<MulticastTimeout>0</MulticastTimeout>
					<DatagramMaximumPacketSize>8192</DatagramMaximumPacketSize>
				</Incoming>
				<Outgoing>
					<ReuseAddress>true</ReuseAddress>
					<ReceiveBufferSize>65000</ReceiveBufferSize>
					<SendBufferSize>65000</SendBufferSize>
					<!-- <MulticastBindToAddress>true</MulticastBindToAddress> -->
					<!-- <MulticastInterfaceAddress>192.168.1.22</MulticastInterfaceAddress> -->
					<!-- <TrafficClass>0</TrafficClass> -->
					<MulticastTimeout>50</MulticastTimeout>
					<DatagramMaximumPacketSize>4096</DatagramMaximumPacketSize>
				</Outgoing>
			</DatagramConfiguration>
			<UnicastIncoming>
				<ProcessorCount>12</ProcessorCount>
			</UnicastIncoming>
			<UnicastOutgoing>
				<ProcessorCount>24</ProcessorCount>
			</UnicastOutgoing>
			<MulticastIncoming>
				<ProcessorCount>12</ProcessorCount>
			</MulticastIncoming>
			<MulticastOutgoing>
				<ProcessorCount>12</ProcessorCount>
			</MulticastOutgoing>
		</RTP>
		<Application>
			<ApplicationTimeout>60000</ApplicationTimeout>
			<PingTimeout>12000</PingTimeout>
			<ValidationFrequency>20000</ValidationFrequency>
			<MaximumPendingWriteBytes>0</MaximumPendingWriteBytes>
			<MaximumSetBufferTime>60000</MaximumSetBufferTime>
		</Application>
		<StartStartupStreams>true</StartStartupStreams>
		<!-- Properties defined here will be added to the IVHost.getProperties() collection -->
		<Properties>
		</Properties>
	</VHost>
</Root>

That how heap memory looks when wowza starts looks bad:

Is it normal?

And as you can see while wowza working there is a lot of memory cached:

top - 00:28:45 up 12:16,  2 users,  load average: 1.31, 17.85, 42.17
Tasks: 116 total,   1 running, 115 sleeping,   0 stopped,   0 zombie
Cpu(s): 20.7%us,  2.9%sy,  0.0%ni, 73.6%id,  2.6%wa,  0.0%hi,  0.3%si,  0.0%st
Mem:  16405628k total, 12422764k used,  3982864k free,     1360k buffers
Swap:        0k total,        0k used,        0k free,  [B]5991896k cached[/B]

At this night we try to set this enviroment settings:

#!/bin/sh
_EXECJAVA=java
#JAVA_OPTS="-Xms4000M"
# maximum Java heap size
JAVA_OPTS="$JAVA_OPTS -Xmx6000M -Xms6000M -Xss128K"
# Uncomment to run server environment (faster), Note: will only work if server VM install, comes with JDL
JAVA_OPTS="$JAVA_OPTS -server"
# Can be a better GC setting to avoid long pauses
JAVA_OPTS="$JAVA_OPTS -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:+UseParNewGC"

And wowza hangs, but at this time with heap size out of memory.

[Peter_Volkov]

Richard this does not look like performance issue as we do have top performance server: 12 Intel® Xeon® CPU E5649 @ 2.53GHz cores and 16Gb of RAM (see xzerth for cpuinfo). So from hardware point of view I don’t see any problems.