Video encoder is stuck, recreating.

Wowza Streaming Engine
1

While transcoding, the return stream contains no data. The log shows “Video encoder is stuck, recreating.”

The card i’m using is a Tesla K10. Decoding seems to go well using CUDA or NVCUVID.
Encoding using NVENC however does not.

nvidia-smi doesn’t show any encoding activity. Using the default wowza encoder is working well.

Do I have to install any additional software? SDK? Or is the card faulty?

Wowza Streaming Engine Version

4.8.0 (build 20200213142111)

Wowza Streaming Engine Manager Version

4.8.0 (build 20200213142111)

Java Information

Java Version

9.0.4

Java VM Version

9.0.4+11

Java Architecture

64

Java Name

Java HotSpot™ 64-Bit Server VM

Java Vendor

Oracle Corporation

Java Home

C:\Program Files\Java\jre-9.0.4

Java Max Heap Size

5612MB

±----------------------------------------------------------------------------+

| NVIDIA-SMI 442.50 Driver Version: 442.50 CUDA Version: 10.2 |

|-------------------------------±---------------------±---------------------+

| GPU Name TCC/WDDM | Bus-Id Disp.A | Volatile Uncorr. ECC |

| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |

|===============================+======================+======================|

| 0 Tesla K10.G1.8GB TCC | 00000000:07:00.0 Off | Off |

| N/A 37C P0 43W / 117W | 173MiB / 4019MiB | 9% Default |

±------------------------------±---------------------±---------------------+

| 1 Tesla K10.G1.8GB TCC | 00000000:08:00.0 Off | Off |

| N/A 36C P8 17W / 117W | 34MiB / 4019MiB | 0% Default |

±------------------------------±---------------------±---------------------+

±----------------------------------------------------------------------------+

| Processes: GPU Memory |

| GPU PID Type Process name Usage |

|=============================================================================|

| 0 5308 C Insufficient Permissions 135MiB |

±----------------------------------------------------------------------------+

a2d8>nvidia-smi dmon

gpu pwr gtemp mtemp sm mem enc dec mclk pclk

Idx W C C % % % % MHz MHz

0 44 37 - 9 1 0 15 2500 745

1 17 36 - 0 0 0 0 324 324

0 44 37 - 8 1 0 15 2500 745

1 17 36 - 0 0 0 0 324 324

0 43 37 - 7 1 0 12 2500 745

1 17 36 - 0 0 0 0 324 324

0 44 37 - 9 1 0 16 2500 745

1 17 36 - 0 0 0 0 324 324

0 44 37 - 5 0 0 9 2500 745

1 17 36 - 0 0 0 0 324 324

0 44 37 - 8 1 0 14 2500 745

1 17 36 - 0 0 0 0 324 324

#Fields: x-severity x-category x-event date time c-client-id c-ip c-port cs-bytes sc-bytes x-duration x-sname x-stream-id x-spos sc-stream-bytes cs-stream-bytes x-file-size x-file-length x-ctx x-comment

WARN server comment 2020-05-31 12:42:55 - - - - - 8.25 - - - - - - - - RTPSessionDescriptionDataProviderBasic.getStreamInfo: SDP file missing: C:/Program Files (x86)/Wowza Media Systems/Wowza Streaming Engine 4.1.0/content/Gelink-Unifi-Video

WARN server comment 2020-05-31 12:42:57 - - - - - 10.366 - - - - - - - - TranscoderWorkerVideoDecoder.handlePacket[defaultVHost:Ipcam1/definst/Ipcam1.stream: decodeIterations:0]: Source stream frame rate could not be calculated. Using default source stream frame rate: 29.97

WARN server comment 2020-05-31 12:42:57 - - - - - 10.461 - - - - - - - - TranscoderWorkerVideoEncoder.setupEncoder[defaultVHost:Ipcam1/definst/Ipcam1.stream:720p]: Error creating encoder: H264

WARN server comment 2020-05-31 12:42:58 - - - - - 10.604 - - - - - - - - TranscoderWorkerVideoEncoder.run[videoEncoderAddFrame][720p]: 4294967295

WARN server comment 2020-05-31 12:42:58 - - - - - 10.605 - - - - - - - - TranscoderWorkerVideoEncoder.run[videoEncoderGetEncodedFrames][720p]: 4294967295

WARN server comment 2020-05-31 12:43:01 - - - - - 13.951 - - - - - - - - TranscoderWorkerVideoEncoder.run[videoEncoderGetEncodedFrames][720p]: Video encoder is stuck, recreating.

WARN server comment 2020-05-31 12:43:01 - - - - - 13.952 - - - - - - - - TranscoderWorkerVideoEncoder.setupEncoder[defaultVHost:Ipcam1/definst/Ipcam1.stream:720p]: Error creating encoder: H264

WARN server comment 2020-05-31 12:43:04 - - - - - 16.605 - - - - - - - - RTPDePacketizerBase.Waiting for RTCP packet. See docs for (Application.xml: RTP/AVSyncMethod and RTP/MaxRTCPWaitTime).

WARN server comment 2020-05-31 12:43:04 - - - - - 16.752 - - - - - - - - RTPDePacketizerBase.Waiting for RTCP packet. See docs for (Application.xml: RTP/AVSyncMethod and RTP/MaxRTCPWaitTime).

WARN server comment 2020-05-31 12:43:06 - - - - - 19.165 - - - - - - - - TranscoderWorkerVideoEncoder.run[videoEncoderGetEncodedFrames][720p]: Video encoder is stuck, recreating.

WARN server comment 2020-05-31 12:43:06 - - - - - 19.166 - - - - - - - - TranscoderWorkerVideoEncoder.setupEncoder[defaultVHost:Ipcam1/definst/Ipcam1.stream:720p]: Error creating encoder: H264

WARN server comment 2020-05-31 12:43:08 - - - - - 20.731 - - - - - - - - RTCPHandler.isTimeSyncReady: Hit MaxRTCPWaitTime synchronizing on system clock.

WARN server comment 2020-05-31 12:43:08 - - - - - 20.76 - - - - - - - - RTCPHandler.isTimeSyncReady: Hit MaxRTCPWaitTime synchronizing on system clock.

WARN server comment 2020-05-31 12:43:11 - - - - - 24.478 - - - - - - - - TranscoderWorkerVideoEncoder.run[videoEncoderGetEncodedFrames][720p]: Video encoder is stuck, recreating.

…

2

Hi @Rutger-Jan Gelink, what are the specs of you input stream (codec, bitrate, fps, etc.), and what does your transcoder template look like (preferably add it here as an attachment rather than copy/paste it directly into your response)?

Also, it looks like your input stream is unstable and/or has timing issues.

3

Hi @Karel Boek-Senior Consultant, thanks for your reply! No matter what incoming stream i’m using, the nvenc encoder does not seem to kick in. In this particular case i’m using a (grandstream) camera board, which is running fine the last 4 years on this wowza server with the following parameters:

1080p / 6 Mbits CBR / 30fps / H264 Main / I-frame interval 20

Since yesterday i’ve added the kepler card to the system.

I’ve attached the transcoder template (click here: https://www.gelink.nu/data/transrate.xml). I made some minor changes on the default wowza transrate template. Currently using only 720p.

I’ve also tried the standard wowza transcode template, with no succes.

Better interpretable view of nvidia-smi

And dmon:

Besides, i’ve tried to transcode an Axis stream, OBS stream, dahua camera. All with the same result.

Click here (https://www.gelink.nu/data/transcoded_stream.JPG) for the overview page of incoming (active) transcoded stream. It shows 526 @ 0.000 bits/s .

Click here (https://www.gelink.nu/data/example_error.txt) for how the error log looks like in the error log after activating the transcoder. As we speak i’ve synced the camera to a NTP Server (I already did but got de-activated).

4

Just to confirm: the reason that the process name says “Insufficient Permissions” is because you ran nvidia-smi as a user and not as root?

First now I realize that you’re using driver version 442.50, while - according to NVidia’s website, the newest version for a Tesla K10 is 440.64. Can you try to download the driver from here: https://www.nvidia.com/Download/driverResults.aspx/158193/en-us

Sidenote: looks like your transcoder template was ported from an older version of Wowza, and it lacks the Scale tag. Compare your template with a default template from latest Wowza; you can copy the Scale section into your template or re-build your template based on their example. That’ll allow you to also let the GPU do the scaling (provided that the GPU works, of course)

If the other driver doesn’t work, then you could test if GPU-based encoding works with different software, to verify if the card and driver is working (see e.g. https://devblogs.nvidia.com/nvidia-ffmpeg-transcoding-guide/)

5

@Karel Boek-Senior Consultant Thanks for your decent replies!

I forgot to mention that i’m running on a Windows Server (2012 r2 datacenter).

Running ffmpeg for a while (on a 2hr / 8GB mp4 file), both decoder and encoder are succesfully being adressed.

ffmpeg did show up in nvidia-smi at that point Wowza was still showing ‘insufficient permissions’. I did assume either that it had to do something with user rights. However, now i’m not sure anymore. Can it be because of Session 0 isolation? I thougt that was not applicable on tesla drivers, so never gave my attention to it.

I found a transcode template for wowza 4.2.0 on https://www.wowza.com/docs/how-to-set-up-and-run-wowza-transcoder-for-live-streaming , does this one include the scale tag?

Based on this template I set NVCUVID as decoder en NVENC for encoding. Still with the same results…

6

If you’re running on Windows, then indeed you must run Wowza in a user environment in order to use the GPU; the Wowza Transcoder cannot use the GPU when running in session 0.

Probably best to test Wowza and GPU first in an UI environment to make sure that the GPU works. If you use RDP, you can decide which session you want to connect to (the “/admin” parameter will connect you to session 0; I assume there’s lots of info on the net that explain this furrher)

And you test with this workaround: https://www.wowza.com/docs/how-to-enable-hardware-accelerated-transcoding-when-running-as-a-windows-service

If none of this works, then maybe better to file a support ticket with support@wowza.com and also check with NVidia how you can make sure that it’s not a hw problem.

7

I have the Telsa K40c card that is giving the same error. Everything worked properly until 4.8 update along with the new java. GPU decoding works. To the original poster… did you get it to work?

8

I have the Telsa K40c card that is giving the same error. Everything worked properly until 4.8 update along with the new java. GPU decoding works. To the original poster… did you get it to work?

9

I have the Telsa K40c card that is giving the same error. Everything worked properly until 4.8 update along with the new java. GPU decoding works. To the original poster… did you get it to work?

10

@Carey Williamson, the problem is still present. I’m now using ffmpeg for the transcoding processes and output it to wowza. Besides, i’ve added a regular GTX1060 and this card works well within the wowza transcoding environment. It looks like, with the new update, wowza lacks supporting the Tesla K series!

11

This comment does not address the issue of this thread at all. thanks

12

this is very aggravating. I have been able to work around this issue through COVID19 by using CPU encoding. Now that my use situation is beginning to get ramped back up, I really need this to be fixed. Have you tried a different version of Java… other than the v9 that is provided by wowza?

13

Ok @Carey Williamson, upon further discussion in the Engine team, we are seeing general failures with the latest Nvidia Drivers (Version 550 and Cuda 11) Rolling back to 440 does seem to help in most cases. Not all versions of the newest Nvidia drivers support cuda10 (which is what we build against right now).

440 support cuda 10.2. 450+ supports cuda 11. It seems like something breaks for us with cuda 11.

This has been tagged as a feature request and we are updating our docs with the following today:

I’m sorry for the inconvenience, please feel free to send a support ticket if you’d like discuss it with the engineers at Wowza.

14

@Rose Power-Wowza Community Manager

I just installed the latest CUDA 10.2 compatible driver available for Server 2019… 443.18… The problem persists… pic

I was using driver v398.75 with wowza 4.7.6 with no issues before I upgraded to 4.8.0 and Java9.

What is the possibility of an issue with Java?

I replied to my old support ticket to continue troubleshooting.

Thanks

15

Sorry @Carey Williamson that is what tech support suggested, but let me get a more specific response to this particular issue for you.

16

Ok. I’m having tech support take another look at this thread. That error “Video encoder is stuck, recreating.” was happening in 4.8.0 for some folks with your scenario, but was resolved after upgrading to 4.8.5, so our tech support team suspect it was a combination of the OS and GPU model with Wowza Streaming Engine 4.8.0.

But, to make sure you get the correct answer, I am confirming with the engineers the only solution is to upgrade to 4.8.5. Be back soon. Appreciate your patience and communication.

17

I guess I should have mentioned that I did upgrade to 4.8.5 back when it was released. The issue persists.

I have a support ticket that was put on hold through COVID19. It is ticket #347197

18

the link in this article clearly says that Tesla k series GPUs do not have console access requirements.

19

more info on my environment… I was running Windows server 2016. Two days ago I did an in-place upgrade to 2019. The issue persists.

20

I see it. Feel free to reply to that if you’d like to reopen that one as opposed to creating a new one.