Wowza memory RAM consumption increasing on

Wowza Streaming Engine
1

Hello,

Please someone help me. I’ve opened a support ticket, but I have some urgency. I have a server running only Vod Streaming applications and the use of memory ram of this server 5 days ago began to increase and almost crashed the server twice, making me have to restart the wowza to prevent it.

I made no changes to the configuration, that was functioning normally. A client of mine, who uses the protocol RTMPS, was having many claims saying that after 3 seeks the video stops. I did some research on the forum and found the solution maxPendingSeekRequests, that I inserted according to code below:

maxPendingSeekRequests

95

Integer

However, this client has few access, are less than 25 concurrent users, so I can’t find any relation to my problem. In total, the peak of concurrent users that my server has reached with all applications VOD is 120 users. I don’t use Live Streaming on it.

Server configs:

Intel Xeon 3220-Quad Core 2.4GHz

8 GB Ram

Four 500 GB SATA II HDs (WD5002ABYS) with RAID10

1Gbps Network

Wowza Media Server 3 Monthly Edition 3.5.2 build3747

java version “1.7.0_09-icedtea” / OpenJDK 64-Bit Server VM (build 23.7-b01, mixed mode)

I don’t use custom modules.

6503 root 20 0 7184m 3.0g 4476 S 8.9 75.5 437:13.76 java -Xmx4000M -server -Djava.net.preferIPv4Stack=true -Dcom.sun.management.jmxremote=true -Dcom.wowza.wms

[root@stream03 /]# free -m

total used free shared buffers cached

Mem: 4094 4071 22 0 1 791

-/+ buffers/cache: 3277 816

Swap: 1023 377 646

Finally, the only strange log was the log of today: http://www.localmidia.com.br/publicidade/wowzamediaserver_error.log

However, I do not know if it is related to my problem, because I’m suffering with it before the errors recorded in this log, and the logs of other days showed no abnormality.

Could anybody help me?

My vhost config: http://www.localmidia.com.br/publicidade/VHost.xml

Setenv.sh: http://www.localmidia.com.br/publicidade/setenv.sh.txt

Same problem: https://www.wowza.com/forums/showthread.php?27723-Wowza-memory-consumption-increasing-on&p=118600#post118600

2

Hi,

If you don’t do any live streaming you should set all the Application.xml/Streams/StreamType to “file” to stop anyone connecting a live source to any of your applications.

Example:

file

It appears this server is being used for live streaming as I can see this in the error log:

LiveStreamPacketizerSmoothStreaming.handlePacket[vale/definst/aovivo]

LiveStreamPacketizers are only used for live streaming so if the application “vale” has a StreamType “default” and was configured with LiveStreamPacketizers a live source could connect and use this application for streaming to RTMP, RTSP and HTTP protocols.

The server hasn’t been tuned and this will make a big difference.

You can follow the tuning guide found here,

https://www.wowza.com/docs/how-to-do-performance-tuning

I see you have changed the RAM but the rest of the tuning still needs doing.

The following error indicates a bad file access ether due broken file, broken disk or poor connectivity to the file (possible on a shared storage area):

MediaReaderH264.directOutput: Problem reading input file: java.lang.ArrayIndexOutOfBoundsException

Jason

3

Carlos,

Are you seeing outOfMemory errors? The “insufficient bandwidth” messages are not by themselves a big problem. This happens when the user of an RTMP client does repeated seeks in the player. It is the bandwidth at the client. The various GC options should be used carefully. If it is not helping, revert.

Richard

4

Janson,

Thanks for your reply. I used a live streaming yesterday just as a test, application vale, so you found this record in the log.

If you don’t do any live streaming you should set all the Application.xml/Streams/StreamType to “file” to stop anyone connecting a live source to any of your applications.

I don’t have any application configured so, I’m sure. I have a few applications, only 10.

I followed all the steps of the performance tuning tutorial. Could you tell me what is missing I do? Are you think this is causing memory RAM consumption increase?

Carlos

5

Thank you Jason. So far, I think this solved, I’ll be monitoring.

6

Jason,

I’m monitoring my server and I think that the problem was not solved. It’s like wowza stay increasing consumption of ram and not impaire later, causing me to have to restart it.

Already activated the Garbage Collection and noticed that the increasing rate decreased, but still happening.

See this log:

http://www.localmidia.com.br/publicidade/wowzamediaserver_access.log

Conf this app:

http://www.localmidia.com.br/publicidade/Application.xml

Check the “seeks” on log. I notice that when the “seek” works fine it’s followed by “unpause”. However, there is some records on log that have multiple seeks repeated, without “unpause”, as if the user had a problem. I use maxPendingSeekRequests 0 exactly to avoid problems, because it turns out that using RTMPS after 4 “seeks” the video stop for “Insufficient Bandwidth 5.4” as has already been registered by others in the forum. “Insufficient Bandwidth [5,4]”

What do I do?

Carlos

7

Richard,

The GC really helped, but not solved, only slowed the increasing of ram. I don’t let get OutOfMemory, I restart the wowza before taking 100% usage.

What I cannot find is the reason for this increase on ram. I imagine it could be related to the RTMPS and maxPendingSeekRequests 0, but not really sure.

How can I identify the reason of this problem Richard? My server is tuned and peak concurrent users is only 150. And even reducing the number of concurrent users over time, the memory RAM is still used.

Thanks!

8

Hello?

Richard,

The GC really helped, but not solved, only slowed the increasing of ram. I don’t let get OutOfMemory, I restart the wowza before taking 100% usage.

What I cannot find is the reason for this increase on ram. I imagine it could be related to the RTMPS and maxPendingSeekRequests 0, but not really sure.

How can I identify the reason of this problem Richard? My server is tuned and peak concurrent users is only 150. And even reducing the number of concurrent users over time, the memory RAM is still used.

Thanks!

9

Richard,

Could you help me? It’s very difficult to have to keep restarting the Wowza.

I saw the fixed bug of Wowza 3.5: ‘Fixed memory leak with RTMPS streaming when proxyType was set to BEST’. I use the version 3.5.2 build3747, but I think this problem hasn’t been fixed.

The GC really helped, but not solved, only slowed the increasing of ram. I don’t let get OutOfMemory, I restart the wowza before taking 100% usage.

What I cannot find is the reason for this increase on ram. I imagine it could be related to the RTMPS and maxPendingSeekRequests 0, but not really sure.

How can I identify the reason of this problem Richard? My server is tuned and peak concurrent users is only 150. And even reducing the number of concurrent users over time, the memory RAM is still used.

Thanks!

10

I am also facing the same problem. The memory ram of wowza increasing very high

Operion,

Do you use RTMPS streaming?

I’m still having trouble too, I sent the logs files by email to support but the only guidance was upgrade the Wowza to the newest version, 3.6.2.

"Zoran Ursulovici, Jun 25 11:21 am (CDT):

Hi Carlos,

There were several issues resolved in our latest Wowza release. Would it be possible to upgrade your Wowza server to the latest build and check if that fixes the memory leak issue? Here is the direct link where you can get instructions on how to upgrade to Wowza 3.6.2: Software updates"

11

Hi,

@Carlos,

Are you still having trouble after upgrading?

@Operion,

Have you tried getting the latest version to see is this resolves your issue?

You can find the installers page here,

Wowza installers

Jason

Jason,

I still could not perform the update. Once it’s done I’ll be posting if it solved my problem, thanks!

12

Hi,

The VHost.xml looks fine now but you haven’t added any other GC setting which may also be a problem.

Try adding the other pre-configured GC setting by removing the “#” from the bottom line.

# Can be a better GC setting to avoid long pauses
#JAVA_OPTS="$JAVA_OPTS -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:NewSize=512m"

This may help free up some more memory for other needed tasks.

Jason

13

Hi,

Please can you open a ticket by sending an email to support@wowza.com with a brief description of the problem and a link to this thread for reference.

Also include a zipped copy of the [Wowza-Install]/conf, [Wowza-Install]/logs and [Wowza-Install]/bin directories for analysis.

Many thanks

Jason

14

Hi,

@Carlos,

Are you still having trouble after upgrading?

@Operion,

Have you tried getting the latest version to see is this resolves your issue?

You can find the installers page here,

Wowza installers

Jason

15

I am also facing the same problem. The memory ram of wowza increasing very high