Error: java.lang.OutOfMemoryError: Java heap space

[Denis_Denis]

Hello,

This bug has bitten our server some times last 30 days. Any idea what could be causing this?

Wowza Media Server 3.1.2.16 build3037

OS - Ubuntu 12.04

RAM - 16 Gb

CPU - Intel E3-1230V2 (Cores 4, Threads 8)

Text of last error this morning

java.lang.OutOfMemoryError: Java heap space

at java.nio.HeapByteBuffer.(HeapByteBuffer.java:57)

at java.nio.ByteBuffer.allocate(ByteBuffer.java:331)

at org.apache.mina.common.SimpleByteBufferAllocator.allocate(Unknown Source)

at org.apache.mina.common.ByteBuffer.allocate(Unknown Source)

at org.apache.mina.transport.socket.nio.SocketIoProcessor.read(Unknown Source)

at org.apache.mina.transport.socket.nio.SocketIoProcessor.process(Unknown Source)

at org.apache.mina.transport.socket.nio.SocketIoProcessor.access$600(Unknown Source)

at org.apache.mina.transport.socket.nio.SocketIoProcessor$Worker.run(Unknown Source)

at org.apache.mina.util.NamePreservingRunnable.run(Unknown Source)

at java.lang.Thread.run(Thread.java:722)

WARN server comment 2013-05-13 04:20:09 - - - - - 585159.535 - - - - - - - - Unexpected exception.

java.lang.OutOfMemoryError: Java heap space

WARN server comment 2013-05-13 04:21:40 - - - - - 585250.66 - - - - - - - - Unexpected exception.

java.lang.OutOfMemoryError: Java heap space

WARN server comment 2013-05-13 04:23:04 - - - - - 585351.49 - - - - - - - - nullUnexpected exception from exceptionCaught handler.

java.lang.OutOfMemoryError: Java heap space

WARN server comment 2013-05-13 04:24:03 - - - - - 585396.621 - - - - - - - - Unexpected exception.

java.lang.OutOfMemoryError: Java heap space

WARN server comment 2013-05-13 04:23:42 - - - - - 585377.031 - - - - - - - - nullUnexpected exception from exceptionCaught handler.

java.lang.NullPointerException

[install-dir]/bin/setenv.bat

_EXECJAVA=java

JAVA_OPTS=“-Xmx8000M”

Uncomment to run server environment (faster), Note: will only work if server VM install, comes with JDL

JAVA_OPTS=“$JAVA_OPTS -server”

Can be a better GC setting to avoid long pauses

JAVA_OPTS=“$JAVA_OPTS -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:NewSize=512m”

Uncomment to log debug garbage collection information

#NOW=$(date +“%Y-%m-%d-%H-%M”)

JAVA_OPTS=“$JAVA_OPTS -verbose:gc -Xloggc:”/usr/local/WowzaMediaServer/logs/gc_$NOW.log" -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintHeapAtGC"

Uncomment to write heap dump on OutOfMemoryError

#JAVA_OPTS=“$JAVA_OPTS -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/usr/local/WowzaMediaServer”

Uncomment to force Java to use specific language settings

#JAVA_OPTS=“$JAVA_OPTS -Duser.language=en -Duser.country=US -Dfile.encoding=Cp1252”

Uncomment to fix multicast crosstalk problem when streams share multicast port

JAVA_OPTS=“$JAVA_OPTS -Djava.net.preferIPv4Stack=true”

WMSAPP_HOME=/usr/local/WowzaMediaServer

WMSCONFIG_HOME=/usr/local/WowzaMediaServer

WMSCONFIG_URL=

export WMSAPP_HOME WMSCONFIG_HOME JAVA_OPTS _EXECJAVA

[install-dir]/conf/VHost.xml

<Root>
	<VHost>
		<HostPortList>
			<HostPort>
				<ProcessorCount>4</ProcessorCount>
				<IpAddress>*</IpAddress>
				<!-- Separate multiple ports with commas -->
				<!-- 80: HTTP, RTMPT -->
				<!-- 554: RTSP -->
				<Port>1935</Port>
				<SocketConfiguration>
					<ReuseAddress>true</ReuseAddress>
					<!-- suggested settings for video on demand applications -->
					<ReceiveBufferSize>65000</ReceiveBufferSize>
					<SendBufferSize>65000</SendBufferSize>
					<!-- suggest settings for low latency chat and video recording applications
					<ReceiveBufferSize>32000</ReceiveBufferSize>
					<SendBufferSize>32000</SendBufferSize>
					-->
					<KeepAlive>true</KeepAlive>
					<!-- <TrafficClass>0</TrafficClass> -->
					<!-- <OobInline>false</OobInline> -->
					<!-- <SoLingerTime>-1</SoLingerTime> -->
					<!-- <TcpNoDelay>false</TcpNoDelay> -->
					<AcceptorBackLog>100</AcceptorBackLog>
				</SocketConfiguration>
				<HTTPStreamerAdapterIDs>cupertinostreaming,smoothstreaming,sanjosestreaming,dvrchunkstreaming</HTTPStreamerAdapterIDs>
				<HTTPProviders>
					<HTTPProvider>
						<BaseClass>com.wowza.wms.http.HTTPCrossdomain</BaseClass>
						<RequestFilters>*crossdomain.xml</RequestFilters>
						<AuthenticationMethod>none</AuthenticationMethod>
					</HTTPProvider>
					<HTTPProvider>
						<BaseClass>com.wowza.wms.http.HTTPClientAccessPolicy</BaseClass>
						<RequestFilters>*clientaccesspolicy.xml</RequestFilters>
						<AuthenticationMethod>none</AuthenticationMethod>
					</HTTPProvider>
					<HTTPProvider>
						<BaseClass>com.wowza.wms.http.HTTPProviderMediaList</BaseClass>
						<RequestFilters>*jwplayer.rss|*jwplayer.smil|*medialist.smil|*manifest-rtmp.f4m</RequestFilters>
						<AuthenticationMethod>none</AuthenticationMethod>
					</HTTPProvider>
					<HTTPProvider>
						<BaseClass>com.wowza.wms.http.HTTPServerVersion</BaseClass>
						<RequestFilters>*</RequestFilters>
						<AuthenticationMethod>none</AuthenticationMethod>
					</HTTPProvider>
				</HTTPProviders>
			</HostPort>
			
			<!-- 443 with SSL -->
			<!--
			<HostPort>
				<ProcessorCount>4</ProcessorCount>
				<IpAddress>*</IpAddress>
				<Port>443</Port>
				<SSLConfig>
					<KeyStorePath>${com.wowza.wms.context.VHostConfigHome}/conf/keystore.cert</KeyStorePath>
					<KeyStorePassword>[password]</KeyStorePassword>
					<KeyStoreType>JKS</KeyStoreType>
					<SSLProtocol>TLS</SSLProtocol>
					<Algorithm>SunX509</Algorithm>
					<CipherSuites></CipherSuites>
					<Protocols></Protocols>
				</SSLConfig>
				<SocketConfiguration>
					<ReuseAddress>true</ReuseAddress>
					<ReceiveBufferSize>24000</ReceiveBufferSize>
					<SendBufferSize>65000</SendBufferSize>
					<KeepAlive>true</KeepAlive>
					<AcceptorBackLog>100</AcceptorBackLog>
				</SocketConfiguration>
				<HTTPStreamerAdapterIDs>cupertinostreaming,smoothstreaming,sanjosestreaming,dvrchunkstreaming</HTTPStreamerAdapterIDs>
				<HTTPProviders>
					<HTTPProvider>
						<BaseClass>com.wowza.wms.http.HTTPCrossdomain</BaseClass>
						<RequestFilters>*crossdomain.xml</RequestFilters>
						<AuthenticationMethod>none</AuthenticationMethod>
					</HTTPProvider>
					<HTTPProvider>
						<BaseClass>com.wowza.wms.http.HTTPClientAccessPolicy</BaseClass>
						<RequestFilters>*clientaccesspolicy.xml</RequestFilters>
						<AuthenticationMethod>none</AuthenticationMethod>
					</HTTPProvider>
					<HTTPProvider>
						<BaseClass>com.wowza.wms.http.HTTPProviderMediaList</BaseClass>
						<RequestFilters>*jwplayer.rss|*jwplayer.smil|*medialist.smil|*manifest-rtmp.f4m</RequestFilters>
						<AuthenticationMethod>none</AuthenticationMethod>
					</HTTPProvider>
					<HTTPProvider>
						<BaseClass>com.wowza.wms.http.HTTPServerVersion</BaseClass>
						<RequestFilters>*</RequestFilters>
						<AuthenticationMethod>none</AuthenticationMethod>
					</HTTPProvider>
				</HTTPProviders>
			</HostPort>
			-->
			
			<!-- Admin HostPort -->
			<HostPort>
				<ProcessorCount>1</ProcessorCount>
				<IpAddress>*</IpAddress>
				<Port>8086</Port>
				<SocketConfiguration>
					<ReuseAddress>true</ReuseAddress>
					<ReceiveBufferSize>16000</ReceiveBufferSize>
					<SendBufferSize>16000</SendBufferSize>
					<KeepAlive>true</KeepAlive>
					<AcceptorBackLog>100</AcceptorBackLog>
				</SocketConfiguration>
				<HTTPStreamerAdapterIDs></HTTPStreamerAdapterIDs>
				<HTTPProviders>
					<HTTPProvider>
						<BaseClass>com.wowza.wms.http.streammanager.HTTPStreamManager</BaseClass>
						<RequestFilters>streammanager*</RequestFilters>
						<AuthenticationMethod>admin-digest</AuthenticationMethod>
					</HTTPProvider>
					<HTTPProvider>
						<BaseClass>com.wowza.wms.http.HTTPServerInfoXML</BaseClass>
						<RequestFilters>serverinfo*</RequestFilters>
						<AuthenticationMethod>admin-digest</AuthenticationMethod>
					</HTTPProvider>
					<HTTPProvider>
						<BaseClass>com.wowza.wms.http.HTTPConnectionInfo</BaseClass>
						<RequestFilters>connectioninfo*</RequestFilters>
						<AuthenticationMethod>admin-digest</AuthenticationMethod>
					</HTTPProvider>
					<HTTPProvider>
						<BaseClass>com.wowza.wms.http.HTTPConnectionCountsXML</BaseClass>
						<RequestFilters>connectioncounts*</RequestFilters>
						<AuthenticationMethod>admin-digest</AuthenticationMethod>
					</HTTPProvider>
					<HTTPProvider>
						<BaseClass>com.wowza.wms.transcoder.httpprovider.HTTPTranscoderThumbnail</BaseClass>
						<RequestFilters>transcoderthumbnail*</RequestFilters>
						<AuthenticationMethod>admin-digest</AuthenticationMethod>
					</HTTPProvider>
					<HTTPProvider>
						<BaseClass>com.wowza.wms.http.HTTPProviderMediaList</BaseClass>
						<RequestFilters>medialist*</RequestFilters>
						<AuthenticationMethod>admin-digest</AuthenticationMethod>
					</HTTPProvider>
					<HTTPProvider>
						<BaseClass>com.wowza.wms.http.HTTPServerVersion</BaseClass>
						<RequestFilters>*</RequestFilters>
						<AuthenticationMethod>none</AuthenticationMethod>
					</HTTPProvider>
				</HTTPProviders>
			</HostPort>
		
		</HostPortList>
		
		<HTTPStreamerAdapters>
			<HTTPStreamerAdapter>
				<ID>smoothstreaming</ID>
				<Name>smoothstreaming</Name>
				<Properties>
				</Properties>
			</HTTPStreamerAdapter>
			<HTTPStreamerAdapter>
				<ID>cupertinostreaming</ID>
				<Name>cupertinostreaming</Name>
				<Properties>
				</Properties>
			</HTTPStreamerAdapter>
			<HTTPStreamerAdapter>
				<ID>sanjosestreaming</ID>
				<Name>sanjosestreaming</Name>
				<Properties>
				</Properties>
			</HTTPStreamerAdapter>
			<HTTPStreamerAdapter>
				<ID>dvrchunkstreaming</ID>
				<Name>dvrchunkstreaming</Name>
				<Properties>
				</Properties>
			</HTTPStreamerAdapter>
		</HTTPStreamerAdapters>
		<HandlerThreadPool>
			<PoolSize>120</PoolSize>
		</HandlerThreadPool>
		<TransportThreadPool>
			<PoolSize>80</PoolSize>
		</TransportThreadPool>
		<IdleWorkers>
			<WorkerCount>4</WorkerCount>
			<CheckFrequency>50</CheckFrequency>
			<MinimumWaitTime>5</MinimumWaitTime>
		</IdleWorkers>
		<NetConnections>
			<ProcessorCount>4</ProcessorCount>
			<IdleFrequency>250</IdleFrequency>
			<SocketConfiguration>
				<ReuseAddress>true</ReuseAddress>
				<ReceiveBufferSize>65000</ReceiveBufferSize>
				<SendBufferSize>24000</SendBufferSize>
				<KeepAlive>true</KeepAlive>
				<!-- <TrafficClass>0</TrafficClass> -->
				<!-- <OobInline>false</OobInline> -->
				<!-- <SoLingerTime>-1</SoLingerTime> -->
				<!-- <TcpNoDelay>false</TcpNoDelay> -->
				<AcceptorBackLog>100</AcceptorBackLog>
			</SocketConfiguration>
		</NetConnections>
		<HTTPTunnel>
			<KeepAliveTimeout>2000</KeepAliveTimeout>
		</HTTPTunnel>
		<Client>
			<ClientTimeout>90000</ClientTimeout>
			<IdleFrequency>250</IdleFrequency>
		</Client>
		<!-- RTP/Authentication/Methods defined in Authentication.xml. Default setup includes; none, basic, digest -->
		<RTP>
			<IdleFrequency>75</IdleFrequency>
			<DatagramConfiguration>
				<Incoming>
					<ReuseAddress>true</ReuseAddress>
					<ReceiveBufferSize>1024000</ReceiveBufferSize>
					<SendBufferSize>65000</SendBufferSize>
					<!-- <MulticastBindToAddress>true</MulticastBindToAddress> -->
					<!-- <MulticastInterfaceAddress>192.168.1.22</MulticastInterfaceAddress> -->
					<!-- <TrafficClass>0</TrafficClass> -->
					<MulticastTimeout>50</MulticastTimeout>
					<DatagramMaximumPacketSize>4096</DatagramMaximumPacketSize>
				</Incoming>
				<Outgoing>
					<ReuseAddress>true</ReuseAddress>
					<ReceiveBufferSize>65000</ReceiveBufferSize>
					<SendBufferSize>65000</SendBufferSize>
					<!-- <MulticastBindToAddress>true</MulticastBindToAddress> -->
					<!-- <MulticastInterfaceAddress>192.168.1.22</MulticastInterfaceAddress> -->
					<!-- <TrafficClass>0</TrafficClass> -->
					<MulticastTimeout>50</MulticastTimeout>
					<DatagramMaximumPacketSize>4096</DatagramMaximumPacketSize>
				</Outgoing>
			</DatagramConfiguration>
			<UnicastIncoming>
				<ProcessorCount>4</ProcessorCount>
			</UnicastIncoming>
			<UnicastOutgoing>
				<ProcessorCount>8</ProcessorCount>
			</UnicastOutgoing>
			<MulticastIncoming>
				<ProcessorCount>4</ProcessorCount>
			</MulticastIncoming>
			<MulticastOutgoing>
				<ProcessorCount>4</ProcessorCount>
			</MulticastOutgoing>
		</RTP>
		<Application>
			<ApplicationTimeout>60000</ApplicationTimeout>
			<PingTimeout>12000</PingTimeout>
			<ValidationFrequency>20000</ValidationFrequency>
			<MaximumPendingWriteBytes>0</MaximumPendingWriteBytes>
			<MaximumSetBufferTime>60000</MaximumSetBufferTime>
		</Application>
		<StartStartupStreams>true</StartStartupStreams>
		<!-- Properties defined here will be added to the IVHost.getProperties() collection -->
		<Properties>
		</Properties>
	</VHost>
</Root>

[install-dir]/bin/wms.sh

ulimit -n 20000

[install-dir]/bin/startup.sh

ulimit -n 20000 > /dev/null 2>&1

/etc/sysctl.conf

fs.file-max=20000

Thanks!

[Denis_Denis]

java version “1.7.0_15”

OpenJDK 64-bit Server VM (build 23.7-b01, mixed mode)

[Denis_Denis]

Hello!

Your server is not tuned. You have allocated more memory but thats about it. Please see the tuning guide here

Tune Wowza Streaming Engine for performance

We’ve tuned Wowza according to this guide: Tune Wowza Streaming Engine for performance

There is the list of changes we’ve made:

1. In Performance Tuning Guide:

“It is generally best to run the most recent Sun Java JDK. It is best to run a 64-bit operating system with the 64-bit Java VM”

Now in our server:

java version “1.7.0_21”

Java™ SE Runtime Environment (build 1.7.0_21-b11)

Java HotSpot™ 64-Bit Server VM (build 23.21-b01, mixed mode)

Ubuntu 12.10 (x86-64)

2. In Performance Tuning Guide:

“It is best to run the -server version of the Java VM”

Now in our [install-dir]/bin/setenv.sh:

JAVA_OPTS=“$JAVA_OPTS -server”

3. In Performance Tuning Guide:

“If you are running the 64-bit version of the Java VM … and have 16 GB or more RAM, we suggest a heap size of 8000 MB”

Now in our [install-dir]/bin/setenv.sh:

JAVA_OPTS=“-Xmx8000M”

4. In Performance Tuning Guide:

“Garbage collection (GC) tuning in Java is a tricky subject. What works best in one server setup may not work in another. Through trial-and-error and customer feedback, we have several suggested approaches”

So, we have 3 variants:

variant a. “We strongly recommend No Additional Settings. The default -server tuning works quite well for many streaming situations, so in most cases this tuning works great without additional settings.”

variant b. “The Concurrent Collector”:

JAVA_OPTS=“$JAVA_OPTS -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:NewSize=512m”

variant c. “Experimental: Garbage-First garbage collector (G1)”:

JAVA_OPTS=“$JAVA_OPTS -XX:+UnlockExperimentalVMOptions -XX:+UseG1GC”

Now using variant b:

In our [install-dir]/bin/setenv.sh:

JAVA_OPTS=“$JAVA_OPTS -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:NewSize=512m”

5. In Performance Tuning Guide:

“For low-latency chat applications, it is best to use smaller socket buffer sizes (16000 bytes for read and write)”

Now in our [install-dir]/conf/VHost.xml:

16000

16000

6. In Performance Tuning Guide:

“On Linux, increase the maximum number of file descriptors.”

Now in our [install-dir]/bin/wms.sh:

ulimit -n 20000

Now in our [install-dir]/bin/startup.sh:

ulimit -n 20000 > /dev/null 2>&1

“On some versions of Linux, there is also a limit in the kernel that may need to be increased.”

Now in our /etc/sysctl.conf:

fs.file-max=20000

7. In Performance Tuning Guide:

“On Linux, switch to using the Anticipatory elevator (as) elevator algorithm.”

Haven’t do it.

8. In Performance Tuning Guide:

“On Linux, mount your disk with the noatime option”

Done.

9. In Performance Tuning Guide:

“The [total-core-count] refers to the total number of CPU cores in your server… If your server supports hyper-threading, then use the total number of threads. In the above example, if hyper-threading is available, then the total number of threads is: 2 (processor) x 4 (cores) x 2 (threads per core) = 16”

Now our CPU is Intel E3-1230V2 (1 processor x 4 cores х 8 threads per core)

Now in our [install-dir]/conf/VHost.xml:

16

1

480

320

16

8

16

8

8

10. In Performance Tuning Guide:

“Running Multiple Virtual Hosts:”

We don’t use it.

11. In Performance Tuning Guide:

“If you are not doing low latency streaming…”

We’ve done it already in step 5.

12. In Performance Tuning Guide:

“If you are having problems with multiple incoming multicast streams interfering with each other on Linux you may need to set the Java property java.net.preferIPv4Stack to true”

Now in our [install-dir]/bin/setenv.sh:

JAVA_OPTS=“$JAVA_OPTS -Djava.net.preferIPv4Stack=true”

What settings we have change to get rid of “OutOfMemory” error?

Note:

When we using variant a from step 4, the bug has bitten our server each ~6 hours. We have dump ~6Gb which can send to your support.

When we using variant b from step 4, the bug has bitten our server each ~7 days.

We can also send to you archives with our logs, conf, heap dump and else useful data to solve this problem.

I would also recommend upgrading to 3.5.2 to make sure any fixes from the version you are running are available to you.

We’ve tried 3.5.2.05 but got similar error each ~6 hours, so we rolled away to 3.1.2.

Thanks!

[Denis_Denis]

Today we’ve removed

java version “1.7.0_15” OpenJDK 64-bit Server VM (build 23.7-b01, mixed mode)

and installed

java version “1.7.0_21”

Java™ SE Runtime Environment (build 1.7.0_21-b11)

Java HotSpot™ 64-Bit Server VM (build 23.21-b01, mixed mode)

[Andrew_Kennedy1]

Hi,

Your server is not tuned. You have allocated more memory but thats about it. Please see the tuning guide here

https://www.wowza.com/docs/general-tuning

I would also recommend upgrading to 3.5.2 to make sure any fixes from the version you are running are available to you.

Andrew.

[Roger_Littin1]

Hi Denis,

Can you please upload the heap dump to somewhere that we can download it and then open ticket at support@wowza.com with a link to the download.

Please also zip up your conf folder and recent logs and also include a reference to this forum post.

Roger.