Access frame data as they are transcoded

Wowza Streaming Engine
1

I am trying to access the raw frame data during transcoding. I have the transcoder setup and working and have build a built a server module project in eclipse which is getting run but I cannot get onAfterDecodeFrame to fire. Here is the code I have so far:

class FrameGrabber implements ITranscoderVideoDecoderNotify
{
	public void onAfterDecodeFrame(TranscoderSessionVideo sessionVideo, TranscoderStreamSourceVideo sourceVideo, long frameCount) {
		getLogger().info("Frame: " + String.valueOf(frameCount));
	}
}

When I receive and record an RTSP stream, there are no errors in the log but I do not see the frame count in the logs as I was expecting. I think I need to add it to the appInstance with something like addLiveStreamVideoDecoderListener but this does not exist. How can I get this code to run after each frame is decoded? What am I missing?

Many thanks.

2

Hi,

So to get this to happen you need to add all the listeners to the appinstance, so

[PHP] appInstance.addLiveStreamTranscoderListener(new MyTranscoderCreateNotifier();

class MyTranscoderCreateNotifier implements ILiveStreamTranscoderNotify

{

public MyTranscoderCreateNotifier ( )

{

}

public void onLiveStreamTranscoderCreate(ILiveStreamTranscoder liveStreamTranscoder, IMediaStream stream)

{

((LiveStreamTranscoder)liveStreamTranscoder).addActionListener(new MyTranscoderActionNotifier());

}

public void onLiveStreamTranscoderDestroy(ILiveStreamTranscoder liveStreamTranscoder, IMediaStream stream)

{

}

public void onLiveStreamTranscoderInit(ILiveStreamTranscoder liveStreamTranscoder, IMediaStream stream)

{

}

}

class MyTranscoderActionNotifier extends LiveStreamTranscoderActionNotifyBase

{

public MyTranscoderActionNotifier( )

{

}

public void onInitStop(LiveStreamTranscoder liveStreamTranscoder)

{

while(true)

{

TranscoderStream transcoderStream = liveStreamTranscoder.getTranscodingStream();

if (transcoderStream == null)

break;

TranscoderSession transcoderSession = liveStreamTranscoder.getTranscodingSession();

TranscoderSessionVideo transcoderVideoSession = transcoderSession.getSessionVideo();

transcoderVideoSession.addFrameListener(new MyTranscoderVideoDecoderNotifyBase());

break;

}

}

}

class MyTranscoderVideoDecoderNotifyBase extends TranscoderVideoDecoderNotifyBase

{

public MyTranscoderVideoDecoderNotifyBase ( )

{

}

public void onBeforeScaleFrame(TranscoderSessionVideo sessionVideo, TranscoderStreamSourceVideo sourceVideo, long frameCount)

{

}

public void onBeforeDecodeFrame(TranscoderSessionVideo sessionVideo, TranscoderStreamSourceVideo sourceVideo, long frameCount)

{

}

public void onAfterDecodeFrame(TranscoderSessionVideo sessionVideo, TranscoderStreamSourceVideo sourceVideo, long frameCount)

{

}

}

[/PHP]

Be warned though, if you process anything within this part it must keep up with decoding otherwise things will stall and the output from the transcoder waiting for your process.

Andrew.

3

Hi,

So in onBeforeScaleFrame you could add a class which implements ITranscoderFrameGrabResult, an example add

[PHP]sourceVideo.grabFrame(new CreateShot(),640,480);[/PHP]

This would create a new snapshot of the frame, and the size. When it does this you can convert the videoframe into a bufferedimage.

[PHP] class CreateShot implements ITranscoderFrameGrabResult

{

public CreateShot ( )

{

}

public void onGrabFrame(TranscoderNativeVideoFrame videoFrame)

{

BufferedImage image = TranscoderStreamUtils.nativeImageToBufferedImage(videoFrame);

if ( image == null )

{

// error message here

}

else

{

// Do something with the result

}

}

}[/PHP]

Andrew.

4

Hi,

Using this method you are using the decoding page, hence TranscoderVideoDecoderNotifyBase

There may be a way to disable the encoding, however it may require very low level API calls and not sure if it would be possible, you could have to dig around in the template controls, so ILiveStreamTranscoderActionNotify. By default the transcoder checks if any encodes are in use then does the decoding, if there arent any then it doesnt do any.

The simplest solution I can suggest is do a very small encoding, so frame of 200x200 and set the bitrate of say 150k. Do not go too low otherwise CPU does go the other way as the transcoder libraries try very hard to get lower.

You can not really save h264 in a FLV container, so no not possible.

Andrew.

5

Hi,

So you will have issues with such a small reference timescale. It may be keyframes it may be how the frames are encoded hard to say. If the stream is being published and then unpublished it may be doing so before the buffer is flushed. I suspect due to the time the transcoder takes to start up, 200-300ms (perhaps more) then you will lose initial packets, or your expectation of initial packets, which may be the cause.

There is no way to detect the last frame in the transcoder, but you could use getPlayPackets() in the IMediaStream object to cound the packets left.

Andrew.

6

Hi,

So the problem here is that when the source goes away the data is flushed, so no more inbound packets so the queues stop, hence the last packets never make it to the transcoder.

The only solution I could think of would be to push more data into the stream, with the correct timecodes, so that it flushes the data when the source disconnects, but the last second is lost rather than any video you wanted.

You could try the Stream Class with PlayLists which may help, you could set an item to play twice , the second to only play 1 second, when done you could shut down the source and this may resolve your issue.

Andrew.

7

Hi,

There isn’t anything that can be done. If the source goes away then the buffer is flushed. The same happens with other elements within Wowza. You may be correct it may not be the best fit for your service.

Andrew

8

Hi,

The setRepeat function is not for repeating a stream that has been sent into Wowza. In order to do what you what it will require significant work, the best route I could recommend would be one of our consultants via the Find A Consultant link at the top of the forums.

Andrew

9

Thanks very much Andrew, I have got this working! :slight_smile:

I would now like to dump the current frame data to a file (ideally in YUV or some other image format). I have looked at sessionVideo and sourceVideo and see popScalerDecodedFrame and grabFrame but these return Java objects and cant see how to access the raw image data.

What is the best way to access the image data of the current frame? I assume onAfterDecodeFrame is the best place to do this.

Many thanks for you help.

10

Thanks again Andrew, this was very helpful and I am now able to access data from the individual frames! :smiley:

Are these frames from the original video received in the stream or the frames produced as a result of transcoding? I don’t actually need the resulting transcoded video, I just need to analysis the frames received before the video has finished streaming.

Is there a way to disable the encoding process performed by the transcoder? I have tried PassThru in the transcoder template but then onBeforeScaleFrame no longer gets called (which is understandable). I would like avoid using the resources needed to re-encode if possible as this is expensive and of no use in this context. Is there a way to get the transcoder to just decode the frames from the stream without then encoding?

Finally, is it possible to get rtp-live-record to store the H264 data in an FLV container by default father then an MP4? I am already receiving RTMP streams from a Flash client which gets saved as an FLV file and I would like to avoid having different file types produces by different clients.

Thanks again for your help, it is very much appreciated!

11

Thanks for this, producing a small transcoded video is working well. I was also able to produce an FLV file using the transcoder and PassThru encoding which if I understand correctly, doesn’t touch the video data and simply repackages it in a different container.

So, I have things nearly working as required but I have one last problem. I am recording a short 5 second video at 15 fps so I should get about 75 frames. However, the transcoder only gives me about 60 frames and when I extract the frames from the recorded video file using FFmepeg I get more like 75 frames.

I did a bit of investigation and started logging the frameCount of onBeforeScaleFrame and onAfterDecodeFrame which produced interesting results. onAfterDecodeFrame is called 15 times before the first onBeforeScaleFrame but they both end at the same time which results in onBeforeScaleFrame being called 15 times less then onAfterDecodeFrame which explains the 15 missing frames. So I changed to using onAfterDecodeFrame which produces the same number of frames as FFmpeg does but the first 15 frames are exactly the same. Looking at the logs, onBeforeDecodeFrame is called 15 times with a sequentially frameCount, then my method logs being called 15 times with a non-sequential frameCount and then they are called alternatively as before.

I can understand that onBeforeScaleFrame may need to wait until the next keyframe before it can best process the set of previous frames which would explain why it starts 15 frames in but I would expect it to carry on for an additional 15 frames after onAfterDecodeFrame has finished but it doesn’t. It feels like it is stopped once the stream finishes when actually it should be called 15 more times because it starts 15 frames later.

I can understand that onAfterDecodeFrame may need to wait until the next keyframe before it can best decode the set of previous frames which would explain why the first 15 frames all get exported at once (in whatever order they return) after the first 15 frames have been received. However, I would not expect them to all be the same so it feels like just the first or last frame of that set is being used for all 15. This would also imply that I would get a burst of frames after each keyframes but this does not happen after the second keyframe.

I wondered if my image export was using too many resources and stalling the transcoder as you suggested so I commented out all code expect for logging but I still see exactly the same effects. Should onBeforeScaleFrame and onAfterDecodeFrame not be called the same number of times for each stream?

Either way I do it, I am losing 1 second which is 20% of the video so it is a problem. Could I be experiencing a bug in Wowza? Why did you suggest using onBeforeScaleFrame rather then onAfterDecodeFrame? How can I ensure that I get access to every frame of the video during transcoding?

Also, is there any way to detect that this is the last frame of the video in the transcoder?

Thanks again for your help.

12

I just upgraded Wowza from 3.6.3 to 3.6.4 in case there was a bugfix that may have helped but I still get exactly the same behaviour.

Still struggling to work out a way to access all frames during the transcoding… :confused:

13

Thanks for your reply Andrew. This got me thinking and I did some more tests. I streamed a video that was easy to identify each frame and I can confirm that we are not missing frames at the beginning. Frame 1 produced by FFmpeg exactly matched frame 1 produced by onAfterDecodeFrame and onBeforeScaleFrame. Frame 2 from FFmpeg exactly matches frame 2 from onBeforeScaleFrame and exactly matches frame 16 from onAfterDecodeFrame so the first 15 repeated frames produced by onAfterDecodeFrame are repeats of frame 1 so we are not loosing any data from the beginning of the stream which is good news.

I tried an experiment and added Thread.sleep(5000) into onLiveStreamTranscoderDestroy and this time many more frames were produced but the last 15 were all like this (which is not what was recorded):

Frame43

The last 15 are all similar but not exactly the same and it didn’t produce the same amount of frames as FFmpeg even with the strange red frames but I think this confirms that the buffer is being flushed or the transcoding is being stopped too early.

Is there any way to stop the buffer being flushed or ensure that transcoding continues all the way to the end of the stream?

I feel like this could be considered as a bug as we have not lost any data but the last second of frames are not being transcoded. I don’t think onAfterDecodeFrame repeating the first frame 15 times would be considered expected behaviour particularly as it does produce the correct number of frames.

This is currently a show stopper for our project so any help you can give will be very much appreciated.

14

Hi Andrew,

OK, so this makes sense and I can understand why it is happening. While I see that your workaround would avoid the problem, it is not ideal for our situation. The reason for this is we need to stream good quality video from mobiles which are potentially on slow connection so the amount of data we send needs to kept to an absolute minimum and adding another second is increasing the data by 20%. We need to process the video frames as close to real-time as possible so we can give the handset the result from the analysis as close to the stream finishing as possible. If each second actually takes 5 seconds to stream, we are making the user wait a lot of extra time for no additional benefit so this would make the user experience worse.

We are currently trying Wowza with a developer licence and it is working well for our needs except for this issue. I can use your workaround for now but we will need to solve this before going live and I want to make sure that we don’t build our architecture based on Wowza if in fact this can’t be solved. If we were to sign up to a full licence with the transcoder, do you think that this could be considered as a bug? I can see that this wont effect the majority of use cases but not flushing the buffer until the transcoder has finished feels like a good improvement. Alternately fixing the repeated frame issue would also solve the problem as that does produce the correct number of frames.

I’m happy to try suggestions if you can think of a way to prevent the buffer from being emptied when the source goes away. I’ve looked at the API but can see anything obvious.

Thanks again! :slight_smile:

15

Hi,

That is a shame, but I can understand. So, in the interest of exploring all the options, I have been trying to stop the source going away by setting the item to place twice. I have added the following code to my module:

[PHP]public void onStreamCreate(IMediaStream stream) {

getLogger().info("onStreamCreate by: " + stream.getClientId());

IMediaStreamActionNotify2 actionNotify = new StreamListener();

stream.addClientListener(actionNotify);

}

class StreamListener implements IMediaStreamActionNotify2 {

public void onPublish(IMediaStream stream1, String streamName, boolean isRecord, boolean isAppend) {

getLogger().info("onPublish: " + stream1.getName());

Stream stream = (Stream)stream1.getClient().getAppInstance().getProperties().getProperty(streamName);

stream.setRepeat(true);

}

}[/PHP]

I can see onPublish appear in the logs so I know it is being called but my attempt to get the stream to repeat doesnt seem to work.

Am I on the right track here? How would I get the stream to repeat before closing down the source as you suggested?

Many thanks.

16

Thanks Andrew.

I will investigate the options and may take this route.

17

I have found a way to slowly pipe (simulating real time) a video to FFmpeg and for it to start writing frames as soon as it starts decoding the video. This is exactly what I need so now I just need to find a way to feed the video to ffmpeg as it is received.

I have looked to see if Wowza writes the video file as it is received but I can’t see any evidence of this, it appears to build the file in memory and then write it all to disk in one go at the end. Is there any way to access the video data as it is received and write it to a file in real time?

Thanks for any help, I feel that I’m close to a solution using Wowza! :slight_smile: