RTMP -> Wowza -> HLS | ABR Segments are drifting apart from one another

I have Flash Media Encoder Live setup to send 3 streams from the same source to Wowza. Wowza is then transmuxing this RTMP stream into HLS segments. However after about 48 hours the playlists of the 3 streams start to drift apart on what segment in the playlist they are on. I am currently using playlists with 8 ten second segments. This is problematic because when playing ABR HLS the player will try to switch to a higher/lower bitrate and if the segments are out of order it will error.

I have been running Wowza for about 6 months and have just started seeing this issue. We are currently sending the most streams we have ever used, but the server doesn’t seem to be bogged down.

Is this a Roku issue?

Richard

What version of Wowza are you using? What is the video source in FMLE?

If you stop then restart the stream is the problem resolved or the same?

Have you set this up?:

https://www.wowza.com/docs/how-to-debug-encoder-multi-bitrate-keyframe-alignment

Roku is involved tho, right. I am assuming from your helpful response here.

There seems to be a correlation (from your report) between this problem and increased amount of streaming, which might not be straining Wowza but may be straining the network, the connection between encoder and Wowza.

Richard

(support note: continues in ticket 11412)

Is there any problem with playback that you can see?

Richard

Thanks for the update - I’m seeing similar problems with Roku using Decklink/Kulabyte. I’ll test on latest and see if it’s fixed any.

Robert,

Any resolutions on this issue ?

We also see audio/video sync issues (not often, but they happen) when playing streams on Roku.

RTMP streaming of the same streams is fine.

No. The segments coming out of the wowza server are the issue. The segments in the playlists should be aligned.

2.2.3 build26454

The video source is a BlackMagic card - component in. If I restart the segments will align but after a few hours will start to drift apart. When i first noticed it a few days had gone by and the one stream was over 2000 behind.

I will look into setting that up.

Yes the end device is a Roku, but I am also testing in quicktime.

Good point on the network traffic between the encoders and wowza server.

I will set that up and get back to this thread also after I take a look at the network traffic.

2011-05-24 12:08:52 EDT comment server INFO 200 - Key frame[lomalinda/definst/4-2]: dts:9949985 pts:19900004 - - - 3305.152 - -

2011-05-24 12:08:52 EDT comment server INFO 200 - Key frame[lomalinda/definst/4-1]: dts:9950019 pts:19900071 - - - 3305.202 - -

2011-05-24 12:08:52 EDT comment server INFO 200 - Key frame[lomalinda/definst/4-3]: dts:9949985 pts:19900004 - - - 3305.202 - -

2011-05-24 12:08:52 EDT comment server INFO 200 - Key frame[lomalinda/definst/4-2]: dts:9950019 pts:19900071 - - - 3305.203 - -

2011-05-24 12:08:52 EDT comment server INFO 200 - Key frame[lomalinda/definst/4-3]: dts:9950019 pts:19900071 - - - 3305.203

Does this indicate a problem or is this just a margin of error in the logger?

After upgrading to the newest Wowza version and have been running for a couple hours everything is looking good. Will update with any progress tomorrow.

Came in this morning to find that the segments have drifted apart again.

2011-05-25 09:03:06 EDT comment server INFO 200 - Key frame[lomalinda/definst/1-1]: dts:79286953 pts:91465076 - - - 78558.937 - -

2011-05-25 09:03:06 EDT comment server INFO 200 - Key frame[lomalinda/definst/1-2]: dts:79286953 pts:91465076 - - - 78558.938 - -

2011-05-25 09:03:06 EDT comment server INFO 200 - Key frame[lomalinda/definst/1-3]: dts:79286953 pts:91465076 - - - 78558.938

That looks correct, however:

1-1

#EXTM3U

#EXT-X-ALLOW-CACHE:NO

#EXT-X-TARGETDURATION:10

#EXT-X-MEDIA-SEQUENCE:15542

#EXTINF:5,

media_15542.ts?wowzasessionid=1194417792

#EXTINF:5,

media_15543.ts?wowzasessionid=1194417792

#EXTINF:5,

media_15544.ts?wowzasessionid=1194417792

#EXTINF:2,

media_15545.ts?wowzasessionid=1194417792

#EXTINF:7,

media_15546.ts?wowzasessionid=1194417792

#EXTINF:4,

media_15547.ts?wowzasessionid=1194417792

#EXTINF:5,

media_15548.ts?wowzasessionid=1194417792

#EXTINF:5,

media_15549.ts?wowzasessionid=1194417792

#EXTINF:5,

media_15550.ts?wowzasessionid=1194417792

#EXTINF:5,

media_15551.ts?wowzasessionid=1194417792

1-2

#EXTM3U

#EXT-X-ALLOW-CACHE:NO

#EXT-X-TARGETDURATION:10

#EXT-X-MEDIA-SEQUENCE:15529

#EXTINF:5,

media_15529.ts?wowzasessionid=1406374313

#EXTINF:5,

media_15530.ts?wowzasessionid=1406374313

#EXTINF:2,

media_15531.ts?wowzasessionid=1406374313

#EXTINF:7,

media_15532.ts?wowzasessionid=1406374313

#EXTINF:4,

media_15533.ts?wowzasessionid=1406374313

#EXTINF:5,

media_15534.ts?wowzasessionid=1406374313

#EXTINF:5,

media_15535.ts?wowzasessionid=1406374313

#EXTINF:5,

media_15536.ts?wowzasessionid=1406374313

#EXTINF:5,

media_15537.ts?wowzasessionid=1406374313

#EXTINF:5,

media_15538.ts?wowzasessionid=1406374313

1-3 was the same as 1-2

This lets you view segment drift of HLS streams: https://github.com/brookemckim/hlspider

I’m seeing this issue on Wowza 3 (the latest). Restarting wowza fixes the issue temporarily. Here is how bad it has become:

— Unaligned with segments : 48751, 48746, 48744 —

Are there any settings or anything I can try to fix the issue, or maybe diagnose what’s going on? If we can, we’d like to avoid transcoding.

Okay, even worse this morning.

— Unaligned with segments : 2178, 1980, 2092 —

The strange thing is, that if I play back any of the individual segments that claim to be lined up, like 2178, 1980, and 2092, they are indeed all showing the same source segments. Here’s the result of curling each of the sub playlists that are out of sync:

#EXTM3U
#EXT-X-VERSION:3
#EXT-X-ALLOW-CACHE:NO
#EXT-X-TARGETDURATION:15
#EXT-X-MEDIA-SEQUENCE:1990
#EXTINF:10,
media-b2564000_1990.ts?wowzasessionid=1049865870
#EXTINF:10,
media-b2564000_1991.ts?wowzasessionid=1049865870
#EXTINF:10,
media-b2564000_1992.ts?wowzasessionid=1049865870
#EXTINF:10,
media-b2564000_1993.ts?wowzasessionid=1049865870
#EXTINF:11,
media-b2564000_1994.ts?wowzasessionid=1049865870
#EXTINF:10,
media-b2564000_1995.ts?wowzasessionid=1049865870
#EXTINF:10,
media-b2564000_1996.ts?wowzasessionid=1049865870
#EXTINF:10,
media-b2564000_1997.ts?wowzasessionid=1049865870
#EXTINF:10,
media-b2564000_1998.ts?wowzasessionid=1049865870
#EXTINF:10,
media-b2564000_1999.ts?wowzasessionid=1049865870

#EXTM3U
#EXT-X-VERSION:3
#EXT-X-ALLOW-CACHE:NO
#EXT-X-TARGETDURATION:15
#EXT-X-MEDIA-SEQUENCE:2102
#EXTINF:10,
media-b1064000_2102.ts?wowzasessionid=171596872
#EXTINF:10,
media-b1064000_2103.ts?wowzasessionid=171596872
#EXTINF:10,
media-b1064000_2104.ts?wowzasessionid=171596872
#EXTINF:10,
media-b1064000_2105.ts?wowzasessionid=171596872
#EXTINF:11,
media-b1064000_2106.ts?wowzasessionid=171596872
#EXTINF:10,
media-b1064000_2107.ts?wowzasessionid=171596872
#EXTINF:10,
media-b1064000_2108.ts?wowzasessionid=171596872
#EXTINF:10,
media-b1064000_2109.ts?wowzasessionid=171596872
#EXTINF:10,
media-b1064000_2110.ts?wowzasessionid=171596872
#EXTINF:10,
media-b1064000_2111.ts?wowzasessionid=171596872

#EXTM3U
#EXT-X-VERSION:3
#EXT-X-ALLOW-CACHE:NO
#EXT-X-TARGETDURATION:15
#EXT-X-MEDIA-SEQUENCE:2188
#EXTINF:10,
media-b464000_2188.ts?wowzasessionid=748396612
#EXTINF:10,
media-b464000_2189.ts?wowzasessionid=748396612
#EXTINF:10,
media-b464000_2190.ts?wowzasessionid=748396612
#EXTINF:10,
media-b464000_2191.ts?wowzasessionid=748396612
#EXTINF:11,
media-b464000_2192.ts?wowzasessionid=748396612
#EXTINF:10,
media-b464000_2193.ts?wowzasessionid=748396612
#EXTINF:10,
media-b464000_2194.ts?wowzasessionid=748396612
#EXTINF:10,
media-b464000_2195.ts?wowzasessionid=748396612
#EXTINF:10,
media-b464000_2196.ts?wowzasessionid=748396612
#EXTINF:10,
media-b464000_2197.ts?wowzasessionid=748396612

These are the three different sub-playlists.

media-b2564000_1990.ts

media-b1064000_2102.ts

media-b464000_2188.ts

are all the same content. How is this possible? The master playlist plays out fine in Quicktime, but it doesn’t seem like it should.

No there is not any that I can see. I’m trying to get it on a player that will switch its bitrate, so I can test if the switching works or not. I can’t imagine it would based on the EXT-X-MEDIA-SEQUENCE number in each of the playlists

I tried adding the chunkBreakOnPTS value as described here: https://www.wowza.com/docs/how-to-debug-encoder-multi-bitrate-keyframe-alignment

After running for about 10 hours, there’s now a drift on one of the three playlists by 4 segments.

Any other ideas?