Wowza and Akamai RTMP push, key frame interval are not constant.

j-c · October 1, 2013, 8:31pm

we are using the wowza server to stream feeds from envivio, harmonic, rgb encoders to akamai “universal streaming”. Akamai reported that the stream being sent by wowza is not of good quality. They found an issue with the keyframe interval of the video packets. The input to Wowza is Mpeg2Ts over UDP. A tcpdump showed that the stream are fine at the wowza input.

two questions:

1\ how can i check the rtmp push stream coming from Wowza?

2\ how can I make wowza to streem with steady keyframe intervals?

Many thanks,

J-C

akamai log output:

Ki_delta=2003

Ki_delta=1502

Ki_delta=1237

Ki_delta=271

Ki_delta=995

Ki_delta=1000

Ki_delta=1496

Ki_delta=507

Ki_delta=1252

Ki_delta=250

Ki_delta=229

Ki_delta=766

Ki_delta=256

Ki_delta=983

Ki_delta=2522

Ki_delta=494

Ki_delta=1758

…

sal_jefferson · October 1, 2013, 2:16pm

Can you please explain your set up in a little more detail?

i.e, any custom modules?

Add ons?

Encoder settings, key frame interval etc.

Salvadore

Sascha_Noesberger · October 3, 2013, 9:50am

Hi Salvadore

I’m working with J-C on this topic. We are using Wowza version 3.6.2 with the PushPublish module version 3.5.

As input we have MPEG2-TS UDP multicast streams. There are no drops or streams that are streaming to the same port at the input of the wowza, we even tested the streams directly at the output of the wowza server (with an other configured application) and with that we were able to stream to a player without any freezes or drops.

Here you have the config of the PushPublishMap.txt file:

harmonicUDPSD.stream={profile:“rtmp-akamai”, streamName:“harmonicUDPSD_1_2700”, akamai.streamId:123456, userName:“123456”, password:“xxxxxx”, sendOriginalTimecodes:true, debugLog:debug, debugPackets:false}

We tried different other variants of that file, with and without “sendOriginalTimecodes:true” and “originalTimecodeThreshold:0x100000” but the outcome was always the same.

As J-C already mentioned the timestamps from the encoders are in a regular interval, but at the akamai side they are completly in different intervals. This is causing problems to akamai for create the chunks for HDS and HLS. Is the wowza server restamping the timestamps? To my understanding with the “sendOriginalTimecodes:true” setting, the wowza should take the original timestamps coming from the encoder. Did we miss something?

We havn’t seen any errors in the logs, here you have a sample from the access logs.

2013-10-03 10:42:19 CEST comment server INFO 200 - LiveReceiver.keyframe[akamai/definst/envivioSD.stream]: dts:85818271 pts:85818351 - - - 46970.865 - - - - – - - - - - - - - - - - - - - - - - - -

2013-10-03 10:42:19 CEST comment server INFO 200 - LiveReceiver.keyframe[akamai/definst/rgbLabSD.stream]: dts:13426947 pts:13427027 - - - 46971.014 - - - - – - - - - - - - - - - - - - - - - - - -

2013-10-03 10:42:20 CEST comment server INFO 200 - LiveReceiver.keyframe[akamai/definst/rgbLabSD.stream]: dts:13427947 pts:13428027 - - - 46971.992 - - - - – - - - - - - - - - - - - - - - - - - -

2013-10-03 10:42:20 CEST comment server INFO 200 - LiveReceiver.keyframe[akamai/definst/harmonicUDPSD.stream]: dts:70345920 pts:70346000 - - - 46972.182 - - - - – - - - - - - - - - - - - - - - - - - -

2013-10-03 10:42:21 CEST comment server INFO 200 - LiveReceiver.keyframe[akamai/definst/envivioSD.stream]: dts:85820271 pts:85820351 - - - 46972.796 - - - - – - - - - - - - - - - - - - - - - - - -

2013-10-03 10:42:21 CEST comment server INFO 200 - LiveReceiver.keyframe[akamai/definst/rgbLabSD.stream]: dts:13428947 pts:13429027 - - - 46973.026 - - - - – - - - - - - - - - - - - - - - - - - -

2013-10-03 10:42:22 CEST comment server INFO 200 - LiveReceiver.keyframe[akamai/definst/envivioSD.stream]: dts:85821391 pts:85821471 - - - 46973.808 - - - - – - - - - - - - - - - - - - - - - - - -

2013-10-03 10:42:22 CEST comment server INFO 200 - LiveReceiver.keyframe[akamai/definst/harmonicUDPSD.stream]: dts:70347920 pts:70348000 - - - 46974.017 - - - - – - - - - - - - - - - - - - - - - - - -

2013-10-03 10:42:22 CEST comment server INFO 200 - LiveReceiver.keyframe[akamai/definst/rgbLabSD.stream]: dts:13429947 pts:13430027 - - - 46974.043 - - - - – - - - - - - - - - - - - - - - - - - -

2013-10-03 10:42:23 CEST comment server INFO 200 - LiveReceiver.keyframe[akamai/definst/envivioSD.stream]: dts:85822271 pts:85822351 - - - 46974.825 - - - - – - - - - - - - - - - - - - - - - - - -

2013-10-03 10:42:23 CEST comment server INFO 200 - LiveReceiver.keyframe[akamai/definst/rgbLabSD.stream]: dts:13430947 pts:13431027 - - - 46975.045 - - - - – - - - - - - - - - - - - - - - - - - -

2013-10-03 10:42:24 CEST comment server INFO 200 - LiveReceiver.keyframe[akamai/definst/harmonicUDPSD.stream]: dts:70349920 pts:70350000 - - - 46975.983 - - - - – - - - - - - - - - - - - - - - - - - -

2013-10-03 10:42:24 CEST comment server INFO 200 - LiveReceiver.keyframe[akamai/definst/rgbLabSD.stream]: dts:13431947 pts:13432027 - - - 46976.049 - - - - – - - - - - - - - - - - - - - - - - - -

2013-10-03 10:42:25 CEST comment server INFO 200 - LiveReceiver.keyframe[akamai/definst/envivioSD.stream]: dts:85824271 pts:85824351 - - - 46976.737 - - - - – - - - - - - - - - - - - - - - - - - -

2013-10-03 10:42:25 CEST comment server INFO 200 - LiveReceiver.keyframe[akamai/definst/rgbLabSD.stream]: dts:13432947 pts:13433027 - - - 46977.015 - - - - – - - - - - - - - - - - - - - - - - - -

2013-10-03 10:42:26 CEST comment server INFO 200 - LiveReceiver.keyframe[akamai/definst/envivioSD.stream]: dts:85825191 pts:85825271 - - - 46977.614 - - - - – - - - - - - - - - - - - - - - - - - -

2013-10-03 10:42:26 CEST comment server INFO 200 - LiveReceiver.keyframe[akamai/definst/rgbLabSD.stream]: dts:13433947 pts:13434027 - - - 46978.026 - - - - – - - - - - - - - - - - - - - - - - - -

2013-10-03 10:42:26 CEST comment server INFO 200 - LiveReceiver.keyframe[akamai/definst/harmonicUDPSD.stream]: dts:70351920 pts:70352000 - - - 46978.158 - - - - – - - - - - - - - - - - - - - - - - - -

2013-10-03 10:42:27 CEST comment server INFO 200 - LiveReceiver.keyframe[akamai/definst/envivioSD.stream]: dts:85826271 pts:85826351 - - - 46978.803 - - - - – - - - - - - - - - - - - - - - - - - -

2013-10-03 10:42:27 CEST comment server INFO 200 - LiveReceiver.keyframe[akamai/definst/rgbLabSD.stream]: dts:13434947 pts:13435027 - - - 46979.042 - - - - – - - - - - - - - - - - - - - - - - - -

2013-10-03 10:42:28 CEST comment server INFO 200 - LiveReceiver.keyframe[akamai/definst/rgbLabSD.stream]: dts:13435947 pts:13436027 - - - 46980.008 - - - - – - - - - - - - - - - - - - - - - - - -

2013-10-03 10:42:28 CEST comment server INFO 200 - LiveReceiver.keyframe[akamai/definst/harmonicUDPSD.stream]: dts:70353920 pts:70354000 - - - 46980.192 - - - - – - - - - - - - - - - - - - - - - - - -

2013-10-03 10:42:29 CEST comment server INFO 200 - LiveReceiver.keyframe[akamai/definst/envivioSD.stream]: dts:85828271 pts:85828351 - - - 46980.967 - - - - – - - - - - - - - - - - - - - - - - - -

2013-10-03 10:42:29 CEST comment server INFO 200 - LiveReceiver.keyframe[akamai/definst/rgbLabSD.stream]: dts:13436947 pts:13437027 - - - 46981.007 - - - - – - - - - - - - - - - - - - - - - - - -

Sascha_Noesberger · October 10, 2013, 9:48am

Adding the sort buffer did help in our case, but we had to double the buffer size to 1500.

Does anyone know what is the unit of this value, packets, ms, …? What are the drawback when increasing this value? Will it lead to additional delay? Does it have an influence on the performance?

For me it is not clear why we need this buffer only when pushing to akamai. We used the same video stream, with an other application on the same server, to stream directly from wowza to the client, this way we hadn’t any problems.

Jason_Hilton · October 3, 2013, 4:46pm

Hi,

This issue is currently being handled in a support ticket and with the more detailed logs it does appear that the stream does have an irregular keyframe interval from the source.

00:00:33 LiveReceiver.keyframe[akamai/_definst_/harmonicUDPSD.stream]: dts:31837920 
00:00:35 LiveReceiver.keyframe[akamai/_definst_/harmonicUDPSD.stream]: dts:31839920 
00:00:37 LiveReceiver.keyframe[akamai/_definst_/harmonicUDPSD.stream]: dts:31841920 
00:00:38 LiveReceiver.keyframe[akamai/_definst_/harmonicUDPSD.stream]: dts:31842320 <--
00:00:39 LiveReceiver.keyframe[akamai/_definst_/harmonicUDPSD.stream]: dts:31843920 
00:00:40 LiveReceiver.keyframe[akamai/_definst_/harmonicUDPSD.stream]: dts:31844080 <--
00:00:42 LiveReceiver.keyframe[akamai/_definst_/harmonicUDPSD.stream]: dts:31845920 
00:00:44 LiveReceiver.keyframe[akamai/_definst_/harmonicUDPSD.stream]: dts:31847920

The keyframe interval was 2 seconds at the beginning of the log and then further down this is 1 second and then it goes back up to a 2 second interval.

The last 3 digits should end 920 if they are every 1 second or even every 2 seconds but this also changes. Please notice the last 3 digits “320” and “080” shown above.

If you think the source has an issue with audio and video alignment, you can add a sort buffer which will allow Wowza to realign the stream using the following guide,

How to fix unaligned video and audio with a server side sort buffer

When the stream timecodes jump back in time this indicates a source issue. We do have an article for troubleshooting messages found in the error log which can be found below followed by the possible causes,

Troubleshooting messages found in the error log

Cause: Generally this indicates an encoder issue. It also could be an issue when the machine CPU utilization or GPU utilization is maximized. To debug, see How to debug AAC or MP3 timecode issues with cupertino packetization.

Try using Advanced Stream Monitor the streams and reset them if they become unhealthy. If you see this message immediately after the stream is published before packetization starts, it can be ignored.

Jason

Jason_Hilton · October 10, 2013, 11:38am

Hi,

Thanks for the update, that’s great news.

The sort buffer is in ms and can be configured to a reasonable number I don’t recommend setting the sort buffer higher than 8000 (8 seconds).

By increasing the sort buffer you will increase the latency on the stream (to the size of the buffer) and use a very small amount of memory.

Most players have their own buffer and it would appear that akamai does not have a buffer and so the packets will need to be sorted before reaching akamai to resolve this issue.

Jason

sal_jefferson · October 2, 2013, 6:59am

Unless you are using Wowza Transcoder Add On, Wowza does not re-encode the stream.

Salvadore

sal_jefferson · October 2, 2013, 7:19am

You can do a simple rtmp push to another Wowza application on the same server and look at that stream.

Salvadore

sal_jefferson · October 2, 2013, 12:06pm

It could be that packets are getting lost in the network before Wowza. Take a look at the Wowza Jitter Buffer. It cannot fix this for mpeg-ts stream but it can log packet loss that is helpful for debuging:

How to turn on an RTP jitter buffer and packet loss logging (RTP and MPEG-TS)

You might be able to fix it with:

How to fix UDP packet loss (MPEG-TS/RTP)

It may also be that you have two encoders sending streams to the same port.

Again, are you getting any error messages in the log? What version of Wowza are you running?

Salvadore

j-c · October 2, 2013, 7:09am

we have the pushpublish addon, the encoders have been tested in (harmonic and RGB):

IBBP, IBP and IP as GOP structur
IDR interval of 1,2 and 4 seconds
bandwidth tested were 100,300,500,900,1500 and 2500 for a SD channel.

All those settings left with freezes on the enduser player and akamai confirmed those keyframe loss.

on the wowza side, we have tested with and without those options" sendOriginalTimecodes:true, originalTimecodeThreshold:0x100000," in the PushPublishMap.txt file; it did not make any difference.

j-c · October 2, 2013, 10:33am

we have enabled the timestamp logging. The input has a keyframe every 2s, which is confirmed by a tcpdump and ffprobe. but wowza logs variable values for keyframes.

logs of wowza:

dts:85674000 pts:85674000

dts:85675000 pts:85675000

dts:85676000 pts:85676000

dts:85677480 pts:85677480

dts:85678000 pts:85678000

dts:85678360 pts:85678360

dts:85680000 pts:85680000

dts:85680440 pts:85680440

dts:85682000 pts:85682000

dts:85683240 pts:85683240

dts:85684000 pts:85684000

dts:85686000 pts:85686000

dts:85688000 pts:85688000

questions:

1\ does Wowza reencode the stream?

2\ how can i check the rtmp push stream coming from Wowza?

3\ how can I make wowza to stream with steady keyframe intervals?

j-c · October 2, 2013, 3:07pm

You can do a simple rtmp push to another Wowza application on the same server and look at that stream.

Salvadore

yes, but would this not be a circle? I mean, if our wowza is badly configured or has something not working correctly. we cannot proof that the wowza stream is fine at that moment. it is confusing. I would like to know a procedure on how to check it, like network dump, payload extraction, and then analysing the content with a tool like ffprobe.

j-c · October 2, 2013, 6:55pm

akamai came back to us and told us that we also have packet that are out of order.

Question: does wowza reorder the frame packets in the stream?

thanks.

Topic		Replies	Views
Wowza 4.0.3 and Akamai Push Wowza Streaming Engine	5	2912	July 21, 2014
Key frame at constant time interval Wowza Streaming Engine	3	4274	November 23, 2015
Encoding and sending keyframe over RTP causes blank/refresh Wowza Streaming Engine	1	1756	September 3, 2015
Issue with RTMP ingest to Akamai Wowza Streaming Engine	3	2622	October 21, 2019
PushPublish to Akamai losing video, becoming an audio-only stream Wowza Streaming Engine	2	2295	September 19, 2014

Wowza and Akamai RTMP push, key frame interval are not constant.

Related topics