Hello,
We are running WowzaStreamingEngine 4.0.4. We are using Java 1.8.0_05, Java Heap Size is set to “Production” level.
We reload Wowza service daily but every few days Wowza service is out of memory even if the number of connection is low. How can we identify what is causing this?
Thank you,
I. P.
Hello there and welcome to the Wowza support forum.
Do you see any errors in the logs?
Do you have any custom modules enabled? If so does removing them solve the issue?
Can you please provide your server specs, and tuning settings? Or are you using the “auto tune” feature?
Is this a dedicated Wowza box? If not, it would help if it were.
Kind regards,
Salvadore
Hi,
This is now being handled in ticket #111073
Daren
Hi,
As you have tuned the server and you’re already using the “G1 (Garbage First) collector” which I would have recommended, please edit the [Wowza-Install]/conf/Tune.xml file to remove the comments “” from around the VMOptions/HeapDumpOnOutOfMemoryError which will create a heapdump when Wowza runs out of memory. The result should look like this.
<VMOption>-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath="${com.wowza.wms.AppHome}/logs"</VMOption>
You will need to save the Tune.xml file after editing and restart Wowza for the changes to take effect.
After you have done this, wait until Wowza runs out of memory and it will create the heapdump which you will need to upload to a sharing site or FTP or web server for download and analysis.
Create a ticket with Wowza support by emailing support@wowza.com
When creating a ticket please attach a ZIP file which contains the following directories and files:
[Wowza-Install]/conf/
[Wowza-Install]/logs/
[Wowza-Install]/transcoder/
[Wowza-Install]/manager/logs/
Please make sure you provide logs which show Wowza server starting.
If you are not sure how to get this information please see the following tutorial.
How to create a compressed zip file in Windows, OS X, and Linux
As for the other message, please see it below followed by the cause.
WARN server comment LiveMediaStreamReceiver.doWatchdog: streamTimeout: Resetting connection
Cause: Wowza can’t connect to a MediaCaster stream. It could be a problem with the network or a problem with the source. For example, the camera was reached, but rejected the connection. Wowza will continue trying to reconnect until the stream is available again.
You can use the How to troubleshoot error messages article for referencing common messages in the error log.
Regards,
Jason
Hello All,
Has there been a conclusion after this point? Is there a known memory leak in Wowza 4?
We are also experiencing a possible memory leak. Should we open a new case or can you update this thread?
Regards,
Bora.
Hi,
There are currently no known memory leaks in Wowza so please do open a support ticket if you are having issues.
Andrew
Hi,
The heap file in that case is generated when the Java JVM experiences an out of memory critical event. You can also trigger a heap dump if this is required. Please see this article which shows a few methods to do this.
Paul
Hello,
I’ve removed comments for
in /conf/Tune.xml
This error occurred again, but I don’t see any new logs except access and error logs. Is this suppose to generate a new log file or are the errors added to existing logs?
Thank you,
Ionatan
I have followed the instructions given here, but I don’t see a heap dump file being created anywhere on my server. Can anyone help?
I have followed the instructions on that article, to automatically create a heap dump on out of memory errors.
This is the error as it appears on the log:
WARN server comment 2016-03-10 22:59:48 - - - - - 641832.52 - - - - - - - - Unexpected exception.: java.lang.OutOfMemoryError: Java heap space|
WARN server comment 2016-03-10 22:59:48 - - - - - 641832.52 - - - - - - - - Unexpected exception.: java.lang.OutOfMemoryError: Java heap space|
And the heapdump file is nowhere to be seen. Is there any other configuration needed?
I did follow that article, and my tune.xml file has this line uncommented:
-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath="${com.wowza.wms.AppHome}/logs"
And still I see no heapdumps on that location.
Hello,
I’ve found this in logfile in the same timeframe when Wowza ran out of memory:
Unexpected exception.: java.lang.OutOfMemoryError: Java heap space|at java.util.HashMap.keySet(HashMap.java:906)|at java.util.HashSet.iterator(HashSet.java:172)|at sun.nio.ch.EPollSelectorImpl.implClose(EPollSelectorImpl.java:144)|at sun.nio.ch.SelectorImpl.implCloseSelector(SelectorImpl.java:113)|at java.nio.channels.spi.AbstractSelector.close(AbstractSelector.java:111)|
Also these errors are in error log
- LiveMediaStreamReceiver.doWatchdog: streamTimeout[…]: Resetting connection: […]
WARN server comment 2014-11-27 11:53:08 - - - - - 42463.098 - - - - - - - -
These are Performance Tuning settings:
Memory
Total: 15.66 GB
Java Heap Size
Production level
10000 MB
Java Garbage Collection Settings
G1 (Garbage First) collector
Hello,
I’ve removed comments for
-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=“${com.wowza.wms.AppHome}/logs”
in /conf/Tune.xml
This error occurred again, but I don’t see any new logs except access and error logs. Is this suppose to generate a new log file or are the errors added to existing logs?
Thank you,
Ionatan
Hi,
You can opt to generate a heap dump manually when you notice the heap increasing/peaking using the commands outlined in that post. Do you also see any hs_err_pid.log files in the bin/ directory?
Another option you can look into is to run a JFR on your Wowza instance. This is built-in to Java 8, and should be available for free when testing.
Michelle
Hello,
In the article How to take a Java heap dump, it mentions that the dump save location should be specified in your JAVA_OPTS command. For example:
-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=[path-to-heapdump]
Where [path-to-heapdump] is the path to where the heap dump file is stored (such as /opt).
Best regards,
Andrew
Hello peers having low heap,
I am trying to narrow down this issue with heap space not getting reclaimed until OOM or service affect the server due to uneffective garbage collection at critical almost full level.
Do you have many or just a few channels INPUT?
Do you suffer signal losses between your headend/encoder and Wowza Streaming Engine (WSE)?
Please check “Incoming Streams” one by one whether uptime is same as application uptime and ideally server uptime (if yes, you would confuse my theory, if no you qualify to actually suffer from this issue)
What input decoder is your most popular?
Do you transcode any channels?
What ouput packetizer is your most popular one?