I have a suspected memory leak in a module I’ve developed which is quite hard to isolate. Essentially my servers max out their memory after pretty much exactly 12 days of running, sending the CPU usage up to 100% and causing Wowza to stop working.
I’m no Java expert, let alone a memory expert, but I can guess where I might be going wrong. One of my custom modules I’ve developed uses the Java class ScheduledThreadPoolExecutor to update a database every 10 seconds and I’m thinking a small inefficiency in the code that gets executed is causing memory to gradually get eaten up. If a connection is established each time the update runs, could that be it? Or perhaps not correctly closing statements? I realise it could be put down to any number of things but given I am a Java novice, knowing the obvious things might help.
I did a debug of free memory using runtime.freeMemory() and it did seem to decrease slightly with every update, but that could well have been something else on my system, I’m just not sure.
Any insight at all would be greatly appreciated. Thanks
without seeing the code it is impossible to tell, however updating a database every 10 seconds seems very heavy and any small memory leak would get noticed very quickly, hence this is a good place to start looking.
Shamrock
Dear MSR123,
We’re using a custom Java module with the same purpose you described in this old post of yours: update a database every 10 seconds. Although we’re using Wowza 3.6.2 in a properly tuned, large capacity server, we’re seeing the same behaviour you described: 100% CPU usage in scenarios in which this wouldn’t definitely be expected.
We’re also trying to pinpoint our custom module code to see if we can find any loopholes on it.
My question to you: did you eventually find out the reason for the memory leak you were experiencing? Your inputs on this could really help us find the solution to our problem as well.
Thanks in advance!
Helder
The best tool in this case is to dump the heap and use a tool such as jhat to analyze the heap. You should be able to look at the object histogram after the server has leaked for a while and see which objects are left in memory.
Charlie
These instance counts are not that unusual. Do you actually have a problem? Are getting OutOfMemory errors?
Java garbage collection will use OS memory and will hold on to it. It can grab all the way up to the max memory you have specified as the max heap size. It will hold on to this memory even when the number of connections to the server goes down. It is best to watch memory using JConsole/JMX to see what Java is actually using internally.
Charlie
It looks like yoiu only have the max heap size set to 1GB. This is tiny. What are the specs of the machine (cpu, memory, 32-bit/64-bit). The tuning guide is here:
https://www.wowza.com/docs/how-to-do-performance-tuning
The server is using very little memory in the image above. So the upward trend does really indicate anything. It is only when it starts to hit close the max heap size and it runs over time that it is at all interesting to look at trends.
Charlie
It doesn’t really work that way. Once it hits a percentage of the max heap size then it will GC to keep it in check. At smaller heap sizes this type of trend really doesn’t mean much.
Charlie
No, 1GB is the right size for a small instance. A couple of potential things to look at:
-
There is a leak (either in Wowza or your custom module). It is best to let the server run until you are sure it is leaking. The take a heap dump and analyze the heap dump. You must do this after the server has been leaking for a while so you can see the leaking objects in the heap.
-
It is not a leak but there is some type of inifinite loop problem. For this you need to analyze stack traces to see where this might be happening. Again, this could be core Wowza or in your custom module. I do not have any other folks reporting inifinite loop problems so I doubt this is the problem.
-
The server is becoming overwhelmed. A small instance can only handle about 150Mbps of traffic and since it is using shared CPU it could be reduced based on what else is going on with the other instance sharing the server. In this case you either need to run more instances or a larger instance.
It is most likely one of these situations.
Charlie
You can’t miss it. It’s one of the charts in the default tab in JConsole.
https://www.wowza.com/docs/how-to-install-and-configure-wowza-streaming-engine#jmx
Richard
A possible change to try
class HealthReporter implements Runnable {
IServer server = null;
public static Connection connRemote = null;
public static PreparedStatement prep = null;
public HealthReporter(IServer server)
{
this.server = server;
try {
connRemote = ConnectionParams.getDBConnection(this.server);
}
catch (SQLException SqlEx)
{
System.out.println(SqlEx.toString());
}
try {
String query = "UPDATE xxx";
prep = connRemote.prepareStatement(query);
}
catch (SQLException ex)
{
System.out.println(ex.toString());
}
}
public void run() {
IOPerformanceCounter ServerPerf = this.server.getIoPerformanceCounter();
double serverOutBitRate = ServerPerf.getMessagesOutBytesRate() * 8;
if ( prep !=null )
{
try {
prep.setDouble(1, serverOutBitRate);
prep.executeUpdate();
}
catch (SQLException ex)
{
System.out.println(ex.toString());
}
}
}
I do not have the mysql classes installed not can not try it, but I hope this gives you the direction to look at ?
Shamrock
try this
class HealthReporter implements Runnable {
IServer server = null;
public static Connection connRemote = null;
public static PreparedStatement prep = null;
public static IOPerformanceCounter ServerPerf = null;
public HealthReporter(IServer server)
{
this.server = server;
try {
connRemote = ConnectionParams.getDBConnection(this.server);
}
catch (SQLException SqlEx)
{
System.out.println(SqlEx.toString());
}
try {
String query = "UPDATE xxx";
prep = connRemote.prepareStatement(query);
}
catch (SQLException ex)
{
System.out.println(ex.toString());
}
}
public void run() {
ServerPerf = this.server.getIoPerformanceCounter();
double serverOutBitRate = ServerPerf.getMessagesOutBytesRate() * 8;
if ( prep !=null )
{
try {
prep.setDouble(1, serverOutBitRate);
prep.executeUpdate();
}
catch (SQLException ex)
{
System.out.println(ex.toString());
}
}
}
do note you could change the variables to private or implement in other ways, but this should remove the large IOPerformance object count.
Shamrock
Thanks. I couldn’t see how to get the objects that are left in memory but after the server was running for a few minutes I got the top instance counts excluding platform. I tried to run jhat again about 15 minutes later and got an out of memory error so there’s got to be something up!
These were the top results, which strike me as a lot of instances.
803 instances of class com.mysql.jdbc.JDBC4DatabaseMetaData
800 instances of class com.mysql.jdbc.JDBC4PreparedStatement
800 instances of class com.mysql.jdbc.PreparedStatement$ParseInfo
309 instances of class com.mysql.jdbc.ConnectionPropertiesImpl$BooleanConnectionProperty
200 instances of class com.mysql.jdbc.JDBC4ResultSet
200 instances of class [Lcom.mysql.jdbc.Field;
To give you an overview of my code, here’s what I’m doing in case there are some obvious errors. I’ve reduced the code for clarity.
public void onServerInit(IServer server) {
stpe = new ScheduledThreadPoolExecutor(1);
HealthReporter health = new HealthReporter(server);
stpe.scheduleAtFixedRate(health, 10, 10, TimeUnit.SECONDS);
}
class HealthReporter implements Runnable {
IServer server = null;
public static Connection connRemote = null;
public HealthReporter(IServer server)
{
this.server = server;
try {
connRemote = ConnectionParams.getDBConnection(this.server);
}
catch (SQLException SqlEx)
{
System.out.println(SqlEx.toString());
}
}
public void run() {
IOPerformanceCounter ServerPerf = this.server.getIoPerformanceCounter();
double serverOutBitRate = ServerPerf.getMessagesOutBytesRate() * 8;
String query = "UPDATE xxx";
PreparedStatement prep = null;
try {
prep = connRemote.prepareStatement(query);
prep.setDouble(1, serverOutBitRate);
prep.executeUpdate();
}
catch (SQLException ex)
{
System.out.println(ex.toString());
}
finally {
if (prep != null)
{
try
{
prep.close();
}
catch (SQLException sqlEx)
{
prep = null;
}
}
}
}
}
I do have prep.close() contained in the finally block already. That should do the same, shouldn’t it?
I tried setting the PreparedStatement and query variables as static class properties instead and then calling prep = connRemote.prepareStatement(query); in the class constructor but that didn’t seem to work after the thread schedule executed it more than twice. What you’re saying makes sense though, I just lack the Java know-how to understand what I can do to free up resources.
Shamrock: just saw your edit. Not sure if your example will work if database connectivity is lost, but I will try it.
The number one instance count from my most recent heap analysis:
1691 instances of class com.wowza.util.IOPerformanceCounter
I have just tried to simulate the effect of memory leakage at high speed. I set the scheduled task to update every millisecond and printed out the free memory. The free memory decreases slowly until after a while jumps back up but to a successively lower maximum each time. Eventually it seems to linger around one figure (30000000) and the real memory of the Java process on my system slowly starts increasing. I really don’t know what to do!
Ok, I’ve tried your suggestion and a few other things and the result is the same. I have run Wowza with the default config with all my modules and libraries removed and the instance count is still just as high. I’m running version 2.2.2 and testing on OS X 1.5.8.
I’ve made a fresh, base install of Wowza 2.2.2 on a test server running Ubuntu and the result is the same. Is this normal? Could someone verify they’re getting the same result? Thanks
pidof java
jmap -dump:format=b,file=wowza.bin 8850
jhat wowza.bin
1143 instances of class com.wowza.util.IOPerformanceCounter
163 instances of class [Lcom.wowza.util.IOPerformanceCounter;
157 instances of class org.apache.commons.modeler.AttributeInfo
120 instances of class com.wowza.wms.vhost.VHostWorkerThread
111 instances of class [Lorg.apache.commons.modeler.ParameterInfo;
110 instances of class com.wowza.wms.application.WMSProperties
Thanks Charlie. It really goes back to my original problem of running out of memory after the 12 day period and not knowing what caused it. Since I was using IOPerformanceCounter in my module and getting the high instance counts, I thought I wasn’t implementing it quite right. Is there any particular memory chart I should monitor in JConsole?
Well, I’ve left it going for an hour or so. The general trend seems to be on the increase. Is this normal? The server is sitting idle.
It’s an m1.small EC2 instance, so I guess the maximum memory is 1.7 GB. From my estimations, if the usage continued increasing according to this trend it would hit 1GB after about 10 days.