But when we transcode it, it chuckle or lag (I don’t know how to describe it better, but picture stops after few secs and then start again). It is very disturbing:
I know that you have already settled this issue with another member of our Support Team, but I wanted to post the recommendations for anyone else that comes across the same issue:
The GPU is being overloaded which is causing the transcoder to skip frames. The transcoder does this to reduce the load when the CPU or GPU requirement is more than what’s available. You can see this in the logs below showing some of your streams skipping frames to the point of only having keyframes.
2017-01-26 00:03:55 CET comment server INFO 200 - TranscodingSession.updateBehindFilter[live/_definst_/slowtv10]: Video behind filter state change. New state: KEYFRAMESONLY
2017-01-26 00:13:01 CET comment server INFO 200 - TranscodingSession.updateBehindFilter[live/_definst_/slowtv7]: Video behind filter state change. New state: KEYFRAMESONLY
2017-01-26 05:22:03 CET comment server INFO 200 - TranscodingSession.updateBehindFilter[live/_definst_/slowtv6]: Video behind filter state change. New state: KEYFRAMESONLY
2017-01-26 07:50:03 CET comment server INFO 200 - TranscodingSession.updateBehindFilter[live/_definst_/slowtv5]: Video behind filter state change. New state: KEYFRAMESONLY
2017-01-26 09:59:54 CET comment server INFO 200 - TranscodingSession.updateBehindFilter[live/_definst_/jvc2.stream]: Video behind filter state change. New state: KEYFRAMESONLY
Below you can see that some of the transcoded streams no longer contain video due to the lack of resources required to transcode them:
2017-01-26 00:22:37 CET comment server INFO 200 - TranscodingSession.updateBehindFilter[live/_definst_/slowtv7]: Video behind filter state change. New state: ALLFRAMESOFF
2017-01-26 11:42:57 CET comment server INFO 200 - TranscodingSession.updateBehindFilter[live/_definst_/slowtv10]: Video behind filter state change. New state: ALLFRAMESOFF
2017-01-26 13:02:28 CET comment server INFO 200 - TranscodingSession.updateBehindFilter[live/_definst_/slowtv2]: Video behind filter state change. New state: ALLFRAMESOFF
This also explains why the bitrate of these streams is much lower than when using the CPU which will likely have not been overloaded or as overloaded during the test performed.
When using the Nvidia SMI to review the load on the GPU(s) this can be very inaccurate when the GPU is being used for transcoding as transcoding uses a fixed point function which is not detected in the Nvidia SMI report and so we recommend reviewing the wowzastreamingengine_access.log file(s) for the " Video behind filter state change. New state: [state]" messages. When overloaded you will first see SKIPXFRAME messages with either a 1, 2 or 4 replacing the X. A small number of SKIP1FRAME followed by ALLFRAMESON indicates you’re at the limitation of the CPU or GPU. If you see any SKIP4FRAME, KEYFRAMESONLY or ALLFRAMESOFF messages this is overloaded which will either result in a poor quality transcoded stream or no video in the stream. These messages are often missed as they’re INFO messages and do not appear in the wowzastreamingengine_error.log file.
To resolve this problem you would need to either reduce the number of streams being transcoded or reduce the number of encodes being created for each stream.
Note: A higher quality encode requires more resources than a lower quality encode. For example a 1080p encode requires more CPU/GPU than a 360p encode created from the same source stream.
@Jason_Hatchett,
i have also this Problem, but we do only CPU-.Transcoding, no GPU. I have the Problem above on 3-4 Machines with a Dual Xeon E5-2630v4 (2,20 Ghz) 20 Cores / 40 Threads / 128GB RAM, 960GB Nvme SSD with Wowza 4.8.15+3 or 4.8.17+1. And also on a Ryzen 9 5950X with 128GB RAM and SSD´s. All Server run with Windows Server 2012, 2016 or 2019.
Wowza Setup is nothing special with Standard-Settings. The Servers are only for Transcoding and HLS-Push to the Akamai-Network. NO deliver to the Endusers. Ingest-Streams are 720p50 / 5Mbit. We make 4 Renderings (600, 480,360 and 240p). The Servers run sporadically into this Problem. If i switch the Ingest-Stream to another Server with same Setup, already is fine. I only noticed the Problem since we updated to Wowza 4.8.14 or higher on this Machines but not previously.
I can’t imagine that the server cpu’s should be overloaded. CPU-Load was with this one Stream only near 10%.
That should not be happening obviously. Please send in a support ticket with all the information on video source, protocol, codecs and attach logs so we can properly test and diagnose the issue.
Also please be sure to include the OS and versions of iOS. Thanks.
