Live video stops working due to exceptions

Wowza Streaming Engine
1

Hi.

I have a strange issue where a live video stops working with following exceptions:

ERROR server comment 2011-04-10 08:41:26 - - - - - 8.032 - - - - - - - - Zero sized function (client:1064043662:FMLE/3.0 (compatible; Lavf52.88.0)): type:20 size:0:

ERROR server comment 2011-04-10 08:41:28 - - - - - 9.478 - - - - - - - - AMFDataMixedArray.deserialize[25:8]: java.nio.BufferUnderflowException

WARN server comment 2011-04-10 08:46:28 - - - - - 309.229 - - - - - - - - LiveStreamPacketizerSmoothStreaming.handlePacket[live/_definst_/70a396f99d15b253.sdp]: Fragment duration greater than suggested range of 1-4 seconds. Adjust keyframe interval accordingly: Fragment durations: [13.3,13.3,13.3]

ERROR server comment 2011-04-10 13:37:43 - - - - - 17784.712 - - - - - - - - processNextRequest: error: java.lang.ArrayIndexOutOfBoundsException:

Is there a known issue, and there any patch available for this?

This happens on latest Wowza 2.2.3.

Thanks.

2

Any update about this issue? The problem goes away once we stop and start the stream, and then it works stable for some time, so I guess it something that accumulates over time.

Thanks.

3

I’m getting the exact same issue and it caused a huge customer s*** storm :frowning:

Error log:

ERROR	server	comment	2011-04-10	04:50:29	-	-	-	-	-	7425872.812	-	-	-	-	-	-	-	-	Zero sized function (client:1645582012:WIN 10,2,154,25): type:20 size:0:
ERROR	server	comment	2011-04-10	04:50:30	-	-	-	-	-	7425873.546	-	-	-	-	-	-	-	-	Zero sized function (client:31286727:WIN 10,2,152,26): type:20 size:0:

Access Log:

2011-04-10	04:00:11	BST	comment	server	INFO	200	-	LivePlayer.play[d6b682c5-191b-419f-b789-0d29d69add7e/_definst_/itu_iec_2011]: Dynamic Stream Markers are on.	-	-	-	7422855.062	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
2011-04-10	04:00:11	BST	comment	server	ERROR	500	-	Zero sized function (client:493575615:WIN 10,2,152,32): type:20 size:0:	-	-	-	7422855.218	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
2011-04-10	04:00:11	BST	comment	server	ERROR	500	-	Zero sized function (client:493575615:WIN 10,2,152,32): type:20 size:0:	-	-	-	7422855.218	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
2011-04-10	04:00:11	BST	comment	server	ERROR	500	-	Zero sized function (client:493575615:WIN 10,2,152,32): type:20 size:0:	-	-	-	7422855.218	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-

All other streams continued as normal.

4

Have you tried patch11. This might help.

I believe this can also happen if there is not enough bandwidth between encoder and Wowza. FMLE will get behind and will send incorrect data.

Charlie

5

I would watch the Wowza server over JMX to see if memory is running up. I would watch the encoding side for the same thing. It could also be sure that the connection between Wowza as the encoder not is the issue. There is no patch for this.

Setup a test with Wowza and FMLE on a local network. I think you will find that it runs and runs and runs.

Charlie

6

Take a look at this thread:

https://www.wowza.com/forums/showthread.php?10036-Zero-sized-function-packetSizeReset

Richard

7

Have you followed this suggestion?:

https://www.wowza.com/forums/showthread.php?10036-Zero-sized-function-packetSizeReset#2

Richard

8

@Sybernow,

Sounds like an issue on the encoder side.

Richard

9

The latest patch (2.2.3.11 at this writing) is always here:

https://www.wowza.com/docs/wowza-streaming-engine-software-updates

Richard

10

Great, thanks for the update.

That WARN line is for Smooth (Silverlight). You can remove the smooth packetizer from LiveStreamPacketizers if you are not streaming to Silverlight.

Richard

11

Might be some useful info here:

https://www.wowza.com/forums/showthread.php?7656-RTMPT-random-disconnect-Zero-sized-function-exception

Richard

12

Hi.

The problem described there is different from what I have.

The thread speaks about streaming MPEGTS, while in my case I’m pushing RTMP stream to Wowza.

It works normally for some time, then purely randomly, on the same stream, it stops working (others work well).

Restarting the streaming (stopping / starting the streaming in encoder) resolves the issue.

Before that, we used RTSP-to-RTMP bridge, and all worked fine.

Are there any know issues in Wowza working with RTMP streams?

Thanks

13

Hi.

  1. I changed the GC settings already after the previous issue of Wowza server completely hanging, and now it doesn’t happen indeed - but it is not related to the problem - the issue happens only with single stream, others work fine.

  2. If this is an encoder side issue, how then other streams are working perfectly without any problem? If I stop and start the encoder for this particular stream, it works fine for some random time, then sporadically gets this exception. There are no errors on the encoder side - moreover pushing RTSP video through another server works just fine - it never stops.

Are there any probing tools available that can tell the status of the stream, without me checking it all the time?

Any other info I can provide you with so you could diagnose the issue?

Thanks.

14

Hi.

  1. I’m using the latest version 2.2.3, where I can download patch 11?

  2. Encoder and Wowza are located on same machine, so this not a bandwidth problem.

  3. I’m not using FMLE, but FFMPEG-based encoding tool.

Again, other streams are working great - this basically this particular stream which gives troubles.

Can Wowza recover from such exception, restart the stream if needed and continue working normally?

Thanks.

15

Thanks, got it.

Will test it and post here whether it solves the issue.

16

An update, while usually it has stopped after 10-15 minutes so far, now it keeps working after several hours of running.

I see a single warning which probably not related, but I would still to know what it means:

WARN server comment 2011-04-20 22:05:58 - - - - - 26.034 - - - - - - - LiveStreamPacketizerSmoothStreaming.handlePacket[live/definst/streamtest]: Fragment duration greater than suggested range of 1-4 seconds. Adjust keyframe interval accordingly: Fragment durations: [5.7,5.3,5.4]

Thanks again.

17

Ok, I confirm that after 12 hours of work, there were not errors at all.

Thanks for the fix - hope to see a production release with it soon. :slight_smile:

About the WARN line, does it mean I will have issues streaming to Silverlight clients? And it can be resolved by lowering the key-frame interval as suggested?

Regards.

18

Happy too early :(.

After 15 hours of working, the exception is back, and that time it killed almost all other streams:

ERROR server comment 2011-04-21 13:06:19 - - - - - 54047.142 - - - - - -

    • Zero sized function (client:1405423478:FMLE/3.0 (compatible; Lavf52.88.0)): type:20 size:0:

ERROR server comment 2011-04-21 13:06:19 - - - - - 54047.143 - - - - - -

    • Zero sized function (client:1405423478:FMLE/3.0 (compatible; Lavf52.88.0)): type:20 size:0:

ERROR server comment 2011-04-21 13:06:19 - - - - - 54047.143 - - - - - -

    • Zero sized function (client:1405423478:FMLE/3.0 (compatible; Lavf52.88.0)): type:20 size:0:

ERROR server comment 2011-04-21 13:06:19 - - - - - 54047.144 - - - - - -

    • AMFDataMixedArray.deserialize[25:23]: java.nio.BufferUnderflowException

ERROR server comment 2011-04-21 13:06:19 - - - - - 54047.146 - - - - - -

    • Zero sized function (client:1405423478:FMLE/3.0 (compatible; Lavf52.88.0)): type:20 size:0:

ERROR server comment 2011-04-21 13:06:19 - - - - - 54047.146 - - - - - -

    • Zero sized function (client:1405423478:FMLE/3.0 (compatible; Lavf52.88.0)): type:20 size:0:

ERROR server comment 2011-04-21 13:06:19 - - - - - 54047.147 - - - - - -

    • Zero sized function (client:1405423478:FMLE/3.0 (compatible; Lavf52.88.0)): type:20 size:0:

What information you need to locate this issue and produce a patch?

Thanks.

19

Here is the full exceptions dump - hope it helps:

wowzamediaserver_access.log:2011-04-21 13:06:19 IDT comment server ERROR 500 - AMFDataMixedArray.deserialize[25:23]: java.nio.BufferUnderflowException - - - 54047.144 - - - - - - - - - - - - -

wowzamediaserver_access.log:2011-04-21 13:22:36 IDT comment server ERROR 500 - AMFData.getReference: java.nio.BufferUnderflowException - - - 14.194 - - - - - - - - - - - - - - - -

wowzamediaserver_access.log:2011-04-21 13:22:37 IDT comment server ERROR 500 - AMFDataMixedArray.deserialize[25:10]: java.nio.BufferUnderflowException - - - 15.989 - - - - - - - - - - - - - -

wowzamediaserver_access.log:2011-04-21 13:22:37 IDT comment server ERROR 500 - constructor AMFDataObj(ByteBuffer)[17:25]: java.nio.BufferUnderflowException - - - 15.99 - - - - - - - - - - - - - -

wowzamediaserver_access.log:2011-04-21 13:22:37 IDT comment server ERROR 500 - constructor AMFDataObj(ByteBuffer)[25:25]: java.nio.BufferUnderflowException - - - 15.99 - - - - - - - - - - - - - -

wowzamediaserver_access.log:2011-04-21 13:22:40 IDT comment server ERROR 500 - AMFDataMixedArray.deserialize[25:13]: java.nio.BufferUnderflowException - - - 19.009 - - - - - - - - - - - - - -

wowzamediaserver_access.log:2011-04-21 14:33:56 IDT comment server ERROR 500 - constructor AMFDataObj(ByteBuffer)[25:25]: java.nio.BufferUnderflowException - - - 2032.363 - - - - - - - - - - - - -

wowzamediaserver_access.log:2011-04-21 14:33:56 IDT comment server ERROR 500 - constructor AMFDataObj(ByteBuffer)[15:25]: java.lang.NegativeArraySizeException - - - 2032.407 - - - - - - - - - - - -

wowzamediaserver_error.log:ERROR server comment 2011-04-21 13:06:19 - - - - - 54047.144 - - AMFDataMixedArray.deserialize[25:23]: java.nio.BufferUnderflowException

wowzamediaserver_error.log:ERROR server comment 2011-04-21 13:22:36 - - - - - 14.194 - - - AMFData.getReference: java.nio.BufferUnderflowException

wowzamediaserver_error.log:ERROR server comment 2011-04-21 13:22:37 - - - - - 15.989 - - - AMFDataMixedArray.deserialize[25:10]: java.nio.BufferUnderflowException

wowzamediaserver_error.log:ERROR server comment 2011-04-21 13:22:37 - - - - - 15.99 - - - constructor AMFDataObj(ByteBuffer)[17:25]: java.nio.BufferUnderflowException

wowzamediaserver_error.log:ERROR server comment 2011-04-21 13:22:37 - - - - - 15.99 - - - constructor AMFDataObj(ByteBuffer)[25:25]: java.nio.BufferUnderflowException

wowzamediaserver_error.log:ERROR server comment 2011-04-21 13:22:40 - - - - - 19.009 - - - AMFDataMixedArray.deserialize[25:13]: java.nio.BufferUnderflowException

wowzamediaserver_error.log:ERROR server comment 2011-04-21 14:33:56 - - - - - 2032.363 - - constructor AMFDataObj(ByteBuffer)[25:25]: java.nio.BufferUnderflowException

wowzamediaserver_error.log:ERROR server comment 2011-04-21 14:33:56 - - - - - 2032.407 - - constructor AMFDataObj(ByteBuffer)[15:25]: java.lang.NegativeArraySizeException

20

I would watch the Wowza server over JMX to see if memory is running up. I would watch the encoding side for the same thing. It could also be sure that the connection between Wowza as the encoder not is the issue. There is no patch for this.

Setup a test with Wowza and FMLE on a local network. I think you will find that it runs and runs and runs.

Charlie

I’m monitoring it with top - the java process never passes 230 MB from 2000 allocated. Also, other streams are working perfectly fine, it’s only this particular stream.

About testing - as said, the encoder and Wowza are running on same machine, so any connection issue is improbable.

The patch did improved things, but still not a full solution.

Thanks.