AV sync issues duing wowza transcoder load testing

Wowza Streaming Engine
1

Hi,

I am trying to perform a load test of wowza transcoder. Below are the details of my setup.

  1. Host used : AWS EC2 C4.8x large instance https://aws.amazon.com/ec2/instance-types/

  2. WowzaStreaming engine version : 4.3.0 build16025

  3. Transcoder(transrate) settings:

1920x1080(Pass through) 2Mbps

1280x720 1.3Mbps

640x360 850Kbps

360x240 350Kbps

  1. Wowza application-‘live-http’ is configured with MPEG-DASH and Apple HLS as playback types.

  2. I am using ffmeg to generate rtsp interleaved stream as an input to wowza. I am using the default sample.mp4 as input file which is provided by wowza streaming engine.

  3. Here is the command I am using

ffmpeg.exe -re -i sample.mp4 -vcodec libx264 -vb 500000 -g 60 -vprofile main -acodec aac -ab 128000 -ar 48000 -ac 2 -strict experimental -f rtsp -rtsp_transport tcp rtsp:///live-http/demo/

  1. I am playing the output HLS stream using HLS.js player.

  2. When I provide more than 3 input rtsp stream to wowza streaming engine, I see AV sync issues, video is not smooth in some of the streams when played through HLS.js player.

  3. Until 3 input streams, the output streams generated by wowza play fine without any issues.

  4. Also when I provided 5 input streams to wowza, all the system metrics such as cpu(~50%), memory(4GB), heap(~1GB), network and disk are all well within the limits.

  5. I also looked at the load numbers published by you guys https://www.wowza.com/docs/wowza-transcoder-performance-benchmark . Since I am just performing transrating and not transcoding, Wowza on a C4.8x large should be supporting upto 8 sessions without any issues.

  6. But my tests indicate that I can only support upto 3 sessions. Can you please let me know what I might be doing wrong and how I could increase the capacity of my wowza server?

  7. Also When I ran just the pass through mode(with transcoder disabled) I could support upto 80 sessions without any issues.

Thanks for your time.

2

Hi,

Thank you for this detailed information, this is very helpful. When using the transcoder in Wowza Streaming Engine the source stream should always be the highest bitrate and resolution stream as scaling up is not going to produce a good quality encode and requires the transcoder will require significantly more resources to try and create the encodes. Due to this, I recommend that you use a different source (encoded at 1920x1080 2Mbps or higher) if you’re wanting to keep the above mentioned encodes during a load test, or change the encodes to be of a lower quality (bitrate and resolution) than the source you’re transcoding.

If you’re still having issues after changing the source or encodes, I recommend that you open a ticket with our support team.

When creating a ticket for our Support team, please provide a ZIP file containing the following directories for analysis:

[Wowza-Install]/conf/

[Wowza-Install]/logs/

[Wowza-Install]/transcoder/

Please make sure you provide logs which show Wowza server starting.

If you are not sure how to get this information please see the following tutorial.

How to create a compressed zip file in Windows, OS X, and Linux

Note that you can also try adding a sort-buffer to the application or sort-buffer to the transcoder if you’re sources are having A/V sync issues.

Regards,

Jason

3

Hi,

This is being handled in support ticket #153867

Daren

4

Hi,

Thank you for the repsonse. As you recommended I am using a source with 1920x1080, 30 fps and 2Mbps bit rate and increased the buffers. I still see the same issue. I have also filed a ticket with wowza support.

I am getting the following errors even when i run just 3 sessions

CupertinoPacketHandler.handlePacket[live-http/_definst_/demo5]: Timecode out of order [video]: 35110542:35110547

If i run 5 or more session. I get these errors along with the timecode errors

TranscodingSession.updateBehindFilter[live-http/_definst_/demo4]: Video behind filter state change. New state: SKIP1FRAME
TranscodingSession.updateBehindFilter[live-http/_definst_/demo4]: Video behind filter state change. New state: ALLFRAMESON
CupertinoPacketHandler.handlePacket[live-http/_definst_/demo5]: Timecode out of order [video]: 35110542:35110547

I did some digging and threads in the forum suggests that wowza instance is being overworked. But i see that the CPU utilization is around 60% on c4.8x large instance for 6 sessions and 30% for 3 sessions.

Does this mean i cannot run even 3 sessions on the wowza(c4.8xL with transcoder enabled)? Is there any config which i need to change prevent this error from happening? Thank you.

Thanks!