This strange case reproduced for only one my client’s server. Server version is 3.6.4 build9641. I have following code
for(String vHostName: (List<String>)server.getVHostList().getVHostNames())
{
IVHost vhost = VHostSingleton.getInstance(vHostName);
if(vhost==null)continue;
for(String appName: (List<String>)vhost.getApplicationNames()){
IApplication application = vhost.getApplication((String) appName);
if(application == null) continue;
for(String appIntanceName : application.getAppInstanceNames()){
IApplicationInstance appIntance = application.getAppInstance(appIntanceName);
if(appIntance == null) continue;
for(IClient client: appIntance.getClients())
{
// Exception: java.util.ConcurrentModificationException|at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:859)|at java.util.ArrayList$Itr.next(ArrayList.java:831)|
for(IMediaStream stream : (List<IMediaStream>)client.getPlayStreams()){
}
}
}
}
}
so it looks like client.getPlayStreams() changed outside the look in different thread. All examples in wowza articles use the same approach and client.getPlayStreams() not locked or syncronized before iteration. why this happends and what should I do to prevent this case? or may be i need to handle it and retry ?
I think to avoid this error, you will want to get the playstreams into a list and utilize the list. Something more like the following:
List<IMediaStream> playStreams = client.getPlayStreams();
Iterator iterPlay = playStreams.iterator();
while (iterPlay.hasNext())
{
IMediaStream stream = (IMediaStream)iterPlay.next();
// .. etc ..
}
I think to avoid this error, you will want to get the playstreams into a list and utilize the list. Something more like the following:
List<IMediaStream> playStreams = client.getPlayStreams();
Iterator iterPlay = playStreams.iterator();
while (iterPlay.hasNext())
{
IMediaStream stream = (IMediaStream)iterPlay.next();
// .. etc ..
}
Matt thank you for reply.
Are you sure about this ?
This is from callstack
Exception: java.util.ConcurrentModificationException
at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:859)
at java.util.ArrayList$Itr.next(ArrayList.java:831)
so construction I use and you proposed are equal. Please ask from Development team why java.util.ConcurrentModificationException can be thrown
Where are you implementing this functionality? Is it, by chance, in a play event handler?
This functionality implemented in separate thread and not in any wowza callback. It checks usage statistics periodically.
Why it’s matter there this functionality implemented ? I saw wowza examples working in separate threads and doing the same
I’m trying to get a contextual feel for whats going on as I’m trying to replicate this locally and haven’t had success in doing so thus far. Are you able to reproduce this in latter versions of Wowza (4+)?
Thanks,
Matt
It’s not very common as our module used for years and this client’s first who found this. I cannot reproduce this case but I think we shouldn’t try to reproduce it but rather check in what cases this problem can occur and what should we do in case we get it.
My guess about why this happens is because rtmp support multiple streams and depending to application clients use wowza client can add new streams during livetime. So let’s consider a case when connected client get new stream when I iterate it’s streams in my thread. I think wowza server doesn’t duplication iterator before return it via api so it’s possible that client add or remove stream during iteration in different thread. Currently I’m working with client to check this guess possibility but could you please work with wowza core development team to get an answer to this case. Currently I see only one possible solution: handle this exception and retry. Could you please propose something else ? Or may be my proposed approach is dangerous in some cases ?
Hi,
You need to synchronize the list of play streams when you are going to iterate through it. The following should work,
for(IClient client: appInstance.getClients()) // appInstance.getClients(); returns a copy of the list so shouldn't need to be synchronized.
{
List<IMediaStream> playStreams = client.getPlayStreams(); // client.getPlayStreams(); returns the live list so does need synchronizing.
synchronized(playStreams)
{
for(IMediaStream stream : playStreams)
{
// do something here
}
}
}
Roger.
Roger,
thanks as always ! So you confirm wowza itself do synchronized(playStreams) before change client.getPlayStreams() ?
Hi,
Yes that is correct. The internal methods synchronize on the actual playStreams list before modifying it so you should do the same.
Roger.
Roger,
thanks a lot for clarifications !
what do you think about exception handling in this case ? so if collection externally modified I’ll get exception handle it and try to get lock and retry then but if it’s not I’ll not block collection for wowza internal use. from my POV such retries more efficient. What do you think ?
Hi,
You need to synchronize the list of play streams when you are going to iterate through it. The following should work,
for(IClient client: appInstance.getClients()) // appInstance.getClients(); returns a copy of the list so shouldn't need to be synchronized.
{
List<IMediaStream> playStreams = client.getPlayStreams(); // client.getPlayStreams(); returns the live list so does need synchronizing.
synchronized(playStreams)
{
for(IMediaStream stream : playStreams)
{
// do something here
}
}
}
Roger.
Hi,
Yes that is correct. The internal methods synchronize on the actual playStreams list before modifying it so you should do the same.
Roger.
Where are you implementing this functionality? Is it, by chance, in a play event handler?
I’m trying to get a contextual feel for whats going on as I’m trying to replicate this locally and haven’t had success in doing so thus far. Are you able to reproduce this in latter versions of Wowza (4+)?
Thanks,
Matt