Lip Sync Issue during LiveStreaming

Wowza Streaming Engine
1

Hello - We run WOWZA Media Server 3 on a 64-bit Windows 2008 R2 high-end server. We use One (1) Encoder from Cisco called the Cisco TelePresence Content Server (TCS). TCS Unicasts to WOWZA using (RTMP / H.264) and WOWZA then Livestream’s it on Multicasts to VLC Client Players using RTP over RTSP / H.264.

We are faced with is a Lip Sycn issue during Live Streaming and the issue occurence is very random. In our case, the Audio plays out first before the Video. Whenever we face the issue, we hard-boot the WOWZA Media Server and everything works well after that.

Quickly wanted check on a couple of things:

A. Is there a way to trap some useful logs when the Lip Sync issue occurs? Is Debug the only method?

B. Which logs should i be looking at?

C. What are the possible ways of troubleshooting Lip Sync issue? FYI, my sortbuffersize is set to 2500.

Looking forward to your response.

Thanks

SG

2

This is an encoder issue, Wowza can only debug

Some players, e.g. Flash RTMP player, are able to correct sync in some cases while others can’t.

Generally we have found this is related to an encoding problem with the AAC. The sample frequency is slightly off. If you take the sample rate and assume 1024 samples per-AAC frame and use this information to generate a timecode (in milliseconds) and compare this to the millisecond timecode of the stream they will slowly drift out of sync. There is a way to turn on debugging in Wowza to see if this is the case.

Charlie

Richard

3

Did you add the Property correctly and re-start Wowza, or make sure the application unloaded?

Have you tried setting the Application.xml /MediaCaster /RTPTransportMode to “interleave”? Not sure if that will help, but worth trying

Richard

4

SG,

debugAACTimecodes goes in the /LiveStreamPacketizer Properties container, which is farther down in the Application.xml. You have it in the /Streams Properties container, which includes a “LiveStreamPacketizers” attribute.

Richard

5

Richard - Thanks for the response. We added the code snippet for AAC timecode issue but unable to see the tc[difference] parameter being listed in the command line, although i do see other information. I have ensured that I have included cupertinostreaming in the LiveStreamPacketizers list and that WOWZA is running as a batch job.

Below I have pasted the command output for your reference.

Am i missing something here?

Thanks

SG

PS. Could you please remove the other duplicate post and ensure that this post is reflected in this forum. For some reason I’m unable to see this post being listed and I’m having to acesss this posts links from the e-mail alert I got.

Configure logging: file:///C:/Program Files (x86)/Wowza Media Systems/Wowza Media Server 3.1.2/conf/log4j.properties

INFO server server-start Wowza Media Server 3 Trial Edition (Expires: Nov 20, 2012) 3.1.2 build1624 -

INFO server comment - Server License Key: SVRT3-XXXXX-XXXXX-XXXXX-XXXXX-HTt4R

INFO server comment - Maximum Connections: Unlimited

INFO server comment - Transcoder Streams Available: Unlimited

INFO server comment - Transcoder Watermark: Yes

INFO server comment - nDVR Available: Yes

INFO server comment - DRM Available: Yes

INFO server comment - Hardware Available Processors: 16

INFO server comment - Hardware Physical Memory: 30399MB/32768MB

INFO server comment - Hardware Swap Space: 35903MB/38910MB

INFO server comment - Max File Descriptor Count: Unlimited

INFO server comment - OS Name: Windows Server 2008 R2

INFO server comment - OS Version: 6.1

INFO server comment - OS Architecture: amd64

INFO server comment - OS CPU: amd64

INFO server comment - Java Name: Java HotSpot™ 64-Bit Server VM

INFO server comment - Java Vendor: Oracle Corporation

INFO server comment - Java Version: 1.7.0

INFO server comment - Java VM Version: 21.0-b17

INFO server comment - Java Spec Version: 1.7

INFO server comment - Java Home: C:\Program Files\Java\jdk1.7.0\jre

INFO server comment - Java Max Heap Size: 3555MB

INFO server comment - Java Architecture: 64

INFO server comment - Java Locale[user.language]: en

INFO server comment - Java Locale[user.country]: US

INFO server comment - Java Locale[user.variant]:

INFO server comment - Java Locale[file.encoding]: Cp1252

INFO server comment - Java Timezone[user.timezone]: America/New_York

INFO server comment - Java Args[0]: -Xmx4000M

INFO server comment - Java Args[1]: -Dcom.sun.management.jmxremote=true

INFO server comment - Java Args[2]: -Dcom.wowza.wms.runmode=standalone

INFO server comment - Java Args[3]: -Dcom.wowza.wms.native.base=win

INFO server comment - Java Args[4]: -Dcom.wowza.wms.ConfigURL=

INFO server comment - Server runmode: standalone

INFO server comment - Server native.platform: win

INFO server comment - Server threads[h/t]: 10/10

INFO server comment - JMX java.rmi.server.hostname: localhost

INFO server comment - JMX bind attempt service:jmx:rmi://localhost:8084/jndi/rmi://localhost:8085/jmxrmi

INFO server comment - JMX password file: C:/Program Files (x86)/Wowza Media Systems/Wowza Media Server 3.1.2/conf/jmxremote.password

INFO server comment - JMX access file: C:/Program Files (x86)/Wowza Media Systems/Wowza Media Server 3.1.2/conf/jmxremote.access

INFO server comment - JMX bind successful

INFO server comment - CMDInterface now listening: [any]:8083

INFO vhost vhost-start defaultVHost -

INFO server comment - defaultVHost threads[h/t]:120/80 home:C:/Program Files (x86)/Wowza Media Systems/Wowza Media Server 3.1.2

INFO vhost comment defaultVHost Bind attempt ([any]:1935:4)

INFO vhost comment defaultVHost Bind successful ([any]:1935)

INFO vhost comment defaultVHost Bind attempt ([any]:8086:1)

INFO vhost comment defaultVHost Bind successful ([any]:8086)

INFO server comment - Server.startShutdownHook: Start server shutdown hook

INFO server comment - Wowza Media Server is started!

INFO server comment - ModuleMulticastPublish.onAppStart[live/definst]

INFO server comment - MulticastPublishAddressMapper.init[live/definst]: startingAddress:239.1.1.0:10000 incMode:address

INFO server comment - MulticastPublishAddressMapper.loadMapFile: entries:1

INFO server comment - ModuleLimitPublishedStreamBandwidth MaxBitrate: 4000

INFO application app-start definst live/definst

INFO rtsp connect 308159323 -

INFO server comment - sortPackets[1]: sortBufferSize:2500

INFO server comment - StreamCreate Name:

INFO stream create - -

INFO rtsp announce 308159323 -

INFO server comment - RTPUDPTransport.bind[live/definst]: 0.0.0.0/0.0.0.0:6970

INFO server comment - RTPUDPTransport.bind[live/definst]: 0.0.0.0/0.0.0.0:6971

INFO server comment - RTPUDPTransport.bind[live/definst]: 0.0.0.0/0.0.0.0:6972

INFO server comment - RTPUDPTransport.bind[live/definst]: 0.0.0.0/0.0.0.0:6973

INFO stream publish ABCStream -

INFO server comment - ModuleMulticastPublish.onStreamPublish[live/definst]: ABCStream

INFO server comment - Stream ‘ABCStream’ BitRate: 0kbs, MaxBitrate:4000

INFO rtsp publish 308159323 -

INFO server comment - UDPTransport.firstPacket: bind:0.0.0.0/0.0.0.0:6970 msg:/10.165.243.139:3392

INFO server comment - UDPTransport.firstPacket: bind:0.0.0.0/0.0.0.0:6972 msg:/10.165.243.139:3393

INFO server comment - RTCPHandler.sendFirstRTCPRR[7157,6971,/10.165.243.139:3396]

INFO server comment - UDPTransport.firstPacket: bind:0.0.0.0/0.0.0.0:6971 msg:/10.165.243.139:3396

INFO server comment - LiveStreamPacketizerCupertino.init[live/definst/ABCStream]: chunkDurationTarget: 10000

INFO server comment - LiveStreamPacketizerCupertino.init[live/definst/ABCStream]: chunkDurationTolerance: 500

INFO server comment - LiveStreamPacketizerCupertino.init[live/definst/ABCStream]: audioGroupCount: 3

INFO server comment - LiveStreamPacketizerCupertino.init[live/definst/ABCStream]: playlistChunkCount:3

INFO server comment - onMetaData By: -1

INFO server comment - MediaStreamMap.getLiveStreamPacketizer: Create live stream packetizer: cupertinostreamingpacketizer:ABCStream

INFO server comment - RTCPHandler.sendFirstRTCPRR[7280,6973,/X.X.X.X:3397]

INFO server comment - CupertinoPacketHandler.startStream[live/definst/ABCStream]

INFO server comment - UDPTransport.firstPacket: bind:0.0.0.0/0.0.0.0:6973 msg:/X.X.X.X:3397

INFO server comment - LiveStreamPacketizerCupertino.handlePacket[live/definst/ABCStream]: Video codec:H264 isCompatible:true

INFO server comment - LiveStreamPacketizerCupertino.handlePacket[live/definst/ABCStream]: Audio codec:AAC isCompatible:true

INFO server comment - LiveStreamPacketizerCupertino.handlePacket[live/definst/ABCStream][mp4a.40.2]: AAC Audio info: {AACFrame: codec:AAC, channels:1, frequency:48000, samplesPerFrame:1024, objectTy

pe:LC}

INFO server comment - onMetaData By: -1

INFO server comment - LiveStreamPacketizerCupertino.handlePacket[live/definst/ABCStream][avc1.66.20]: H.264 Video info: {H264CodecConfigInfo: codec:H264, profile:Baseline, level:2.0, frameSize:1280x

720, displaySize:1280x720, PAR:1:1}

INFO server comment - Stream ‘ABCStream’ BitRate: 2985kbs, MaxBitrate:4000

INFO server comment - LiveStreamPacketizerCupertino.endChunkTS[live/definst/ABCStream]: Add chunk: id:1 mode:TS a/v/k:39/61/1 duration:2496

INFO server comment - ModuleMulticastPublish.startStream[live/definst]: {RTPDestination: name: “ABCStream”, audio: 239.255.70.1:10002, video: Y.X.X.X:10000, ttl: 63, isRTPWrapped: true}

INFO rtsp describe 1789179757 -

INFO stream play ABCStream -

INFO server comment - Play Stream: ABCStream

INFO rtsp play 1789179757 -

INFO server comment - Stream ‘ABCStream’ BitRate: 2856kbs, MaxBitrate:4000

INFO server comment - LiveStreamPacketizerCupertino.endChunkTS[live/definst/ABCStream]: Add chunk: id:3 mode:TS a/v/k:158/304/5 duration:10110

INFO server comment - Stream ‘ABCStream’ BitRate: 2842kbs, MaxBitrate:4000

INFO server comment - LiveStreamPacketizerCupertino.endChunkTS[live/definst/ABCStream]: Add chunk: id:4 mode:TS a/v/k:158/302/5 duration:10131

INFO server comment - Stream ‘ABCStream’ BitRate: 2826kbs, MaxBitrate:4000

INFO server comment - Stream ‘ABCStream’ BitRate: 2835kbs, MaxBitrate:4000

INFO server comment - LiveStreamPacketizerCupertino.endChunkTS[live/definst/ABCStream]: Add chunk: id:5 mode:TS a/v/k:158/305/5 duration:10140

INFO server comment - Stream ‘ABCStream’ BitRate: 2848kbs, MaxBitrate:4000

INFO server comment - Stream ‘ABCStream’ BitRate: 2848kbs, MaxBitrate:4000

INFO server comment - Stream ‘ABCStream’ BitRate: 2849kbs, MaxBitrate:4000

INFO server comment - LiveStreamPacketizerCupertino.init[live/definst/ABCStream]: chunkDurationTarget: 10000

INFO server comment - MediaStreamMap.removeLiveStreamPacketizer[live/definst/ABCStream]: Destroy live stream packetizer: cupertinostreamingpacketizer

INFO server comment - LiveStreamPacketizerCupertino.endChunkTS[live/definst/ABCStream]: Add chunk: id:6 mode:TS a/v/k:158/302/5 duration:10089

INFO server comment - LiveStreamPacketizerCupertino.init[live/definst/ABCStream]: chunkDurationTolerance: 500

INFO server comment - LiveStreamPacketizerCupertino.init[live/definst/ABCStream]: audioGroupCount: 3

INFO server comment - LiveStreamPacketizerCupertino.init[live/definst/ABCStream]: playlistChunkCount:3

INFO server comment - MediaStreamMap.getLiveStreamPacketizer: Create live stream packetizer: cupertinostreamingpacketizer:ABCStream

INFO server comment - CupertinoPacketHandler.startStream[live/definst/ABCStream]

INFO server comment - LiveStreamPacketizerCupertino.handlePacket[live/definst/ABCStream]: Video codec:H264 isCompatible:true

INFO server comment - LiveStreamPacketizerCupertino.handlePacket[live/definst/ABCStream]: Audio codec:AAC isCompatible:true

INFO server comment - LiveStreamPacketizerCupertino.handlePacket[live/definst/ABCStream][avc1.66.20]: H.264 Video info: {H264CodecConfigInfo: codec:H264, profile:Baseline, level:2.0, frameSize:1280x

720, displaySize:1280x720, PAR:1:1}

INFO server comment - LiveStreamPacketizerCupertino.handlePacket[live/definst/ABCStream][mp4a.40.2]: AAC Audio info: {AACFrame: codec:AAC, channels:1, frequency:48000, samplesPerFrame:1024, objectTy

pe:LC}

INFO server comment - Stream ‘ABCStream’ BitRate: 2849kbs, MaxBitrate:4000

INFO server comment - Stream ‘ABCStream’ BitRate: 2847kbs, MaxBitrate:4000

INFO server comment - LiveStreamPacketizerCupertino.endChunkTS[live/definst/ABCStream]: Add chunk: id:7 mode:TS a/v/k:159/305/5 duration:10140

INFO server comment - Stream ‘ABCStream’ BitRate: 2847kbs, MaxBitrate:4000

6

Richard - Thanks for the response once again.

  • Below please find the Application.xml , for your reference, where i have included the AACTimer debug property.

live

${com.wowza.wms.context.VHostConfigHome}/content

${com.wowza.wms.context.VHostConfigHome}/keys

cupertinostreamingpacketizer, cupertinostreamingrepeater

sortPackets

true

Boolean

sortBufferSize

2500

Integer

debugAACTimecodes

true

Boolean

  • I have tried hard-booting the server and still don’t find the tc parameter show-up in the command line window.

  • Looks like by default the Mediacaster/RTP/RSTP/RTPTransportMode is set to interleave.

    interleave

    Am i missing anything else?

    Thanks

    SG

7

Dah!!! Sorry about that and many thanks for catching it!

We are now able to see tc in debug output.

While we are trying to recreate the issue, which is pretty random in nature, we are also suspecting integration of the external camera (along with longer HDMI cables) into the Cisco TelePresence C60 Video codec, which is where our content originates before hitting the TCS Encoder.

Many Thanks,

SG

8

Richard - We finally were able to recreate the Lip Sync issue. Looks like it is NOT the external camera or the longer HDMI cables. We are now suspecting the Encoder and/or even the network.

Quickly wanted to understand the difference betweeen tc[0] and tc[1] values. When there is no Lip Sync issue observed, I only see tc[0] in the debug logs. However when Lip Sync issue is experienced I’m able to see two (2) lines of tc[0] and one (1) line of tc[1] in the debug logs and this pattern repeating itself.

Although the reference link states that both tc[0] and tc[1] are acceptable, just wanted to understand the difference between the two, to see if it will provide any hints.

Thanks

SG