Streams for AXIS cameras now dropping frames intermittently

Wowza Streaming Engine
1

Hi all.

We’re running Wowza Streaming Engine 4.2.0 on Linux. We’ve got 4 RTSP live streams from 4 AXIS cameras, and we are recording one of those streams.

Last week, we noticed that 3 of the cameras starting dropping frames. Every so often (perhaps every 2 hours?), the video freezes for ~ 1/4 second, then resumes for 1/2 sec, then freezes, etc. After a varying number of minutes, the freezing stops and the video looks perfect.

Two of the streams are 720p; the other two are 1080p.

Each of the stream URIs is very straightforward, i.e., rtsp://nameofcamera.wellesley.edu/axis-media/media.amp

We’re using Flowplayer 6.0.3 (HTML5) to play the HLS streaming playlist.m3u8

There are usually ~35 connections to the server. CPU usage is ~ 20%. Wowza Heap is ~45%. Total Memory is about 95% (3.6 GB) Total Disk is ~ 20%. Bits out on the Network maxes out at ~ 80Mbps.

Any ideas on how we should troubleshoot? Any data points about our server or setup I forgot to include?

Thanks…k

2

Hello,

The symptoms you describe could be caused by an intermittent network issue. If the timing is precise, exactly every 2 hours, then you can look into any similarly timed processes that could be running on the same network that would be using up the available bandwidth and causing an interruption of the signal from the cameras in question.

One term to search for in your logs would be “UNPUBLISH” around the same time frame that you are seeing these cameras dropping frames, this would be a clear sign of a network interruption between the camera and your Wowza Streaming Engine instance.

Since these camera feeds are recovering, this does not appear to be an issue with Wowza Streaming Engine.

Jason

3

Thanks so much, Jason.

Should I be looking in Engine server logs, or Engine manager logs for “Unpublish”? Or would that word potentially be appearing in both logs?

4

I’m not seeing “unpublish” in the server access logs. There are a lot of entries that say Destroy. These are followed by entries that say Disconnect, then ones that say Stop.

For example, in the 30 minutes between midnight last night and 12:30am, there were 9 entries that said Destroy for camera A; 3 entries that said Destroy for camera B; and 2 entries apiece that said Destroy for cameras C and D.

When I checked on the camera this morning (every 90 minutes for roughly a minute), only once did I see the dropped frames.

What conclusions would you draw from this data?

5

One more question…Our network engineer just asked me to verify…

When the Wowza Streaming Engine is ingesting the rtsp video stream from the AXIS camera and sending it out as an Apple HLS stream, is it end-to-end TCP/IP?

Thanks again…k

6

Hi Jason. Thanks for your thoughts about possible signs of bandwidth loss.

I looked at the article on troubleshooting RTSP/RTP playback. I was a bit confused. The article seemed to focus on troubleshooting streaming to mobile devices. We’re not concerned with mobile devices. Our concern is desktop systems. They’re viewing these streams in Flowplayer HTML5. (See www.wellesley.edu/ravencam for an example.)

Is there a different article that’s more related to our issue, or were there key bits in the article you mentioned that I overlooked?

Thanks…k

7

Hello Kenny,

You should look for these entries in your server access logs. Another term to look for that would follow would be “destroy” (not necessarily directly after but for the .stream file in question)

Following these entries you should see the “create” and “publish” entries as the connection is re-established .

Jason

8

Hello Kenny,

The destroy without an “unpublish” could show signs of a loss of bandwidth, and it may be that these are recovering before the time to live on the session runs out so the “unpublish” entries are not posting.

I want to be sure you are aware of this article as it may help in further troubleshooting the issue you are facing:

https://www.wowza.com/docs/how-to-troubleshoot-rtsp-rtp-playback

To answer the question from your Network Engineer, in the simplest terms this particular protocol will first try UDP delivery and then fallback to TCP delivery if UDP fails.

There is a lot more detail in the article I have mentioned.

If these issues continue to persist, please submit your conf/ and logs/ directories to our Support Team by opening a ticket on this issue.

Jason

9

Hello Kenny,

The main section that is useful here is on turning on RTSP debugging:

https://www.wowza.com/docs/how-to-troubleshoot-rtsp-rtp-playback

Once that has been turned on, if you have an active Maintenance and Support contract it is probably best to submit your conf/ and logs/ directories to our Support Team by opening a ticket on this issue.

Thank you

Jason