Problem Switching HLS Streams

Hi all.

I’m trying to write a plugin that will allow us to switch live streams, initially to insert a looping ‘paused’ stream, but in the limit to allow swapping between other input stream sources. I’ve set things up as per: https://www.wowza.com/docs/how-to-switch-streams-using-stream-class-streams I’ve got one input (RTMP) source and an MP4 as separate playlists which I can swap between. When I deploy this, it works great for RTMP clients, swapping between the sources is quick and seamless.

However, for clients viewing over HLS, the experience isn’t so great. When swapping between playlists, the HLS player (Quicktime or an iPad 3) sometimes freezes for up to 7-8 seconds, and often ‘flickers’ between the two input streams for what appears to be about a segment (10s or so.) I understood there can be problems with transitions if the streams are different dimensions, bitrates, so I’ve tried to ensure they are the same. The MP4 is a result of a webcam recording from Flash into Wowza, as is the live stream. The load in the machine running Wowza is low:

top - 16:34:50 up 1 day, 22:52,  3 users,  load average: 0.04, 0.17, 0.14

So I don’t think it’s running out of resource.

Do you have any ideas as to what could be going on?

Thanks,

Henry

FWIW, the code for my plugin is:

public class StreamSwitchWowzaHTTPProvider extends HTTProvider2Base {
	@Override
	public void onHTTPRequest(IVHost vHost, IHTTPRequest httpRequest, IHTTPResponse httpResponse) {
		
		if (!authenticate(httpRequest)) {
			writeError(403, httpResponse, "Cannot authenticate request");
			return;
		}
		try {
			String action = readRequiredParameter(httpRequest, "action");
			
			switch (action) {
				case "setup":
					setupLiveStreams(vHost, httpRequest, httpResponse);
					break;
				
				case "switch":
					switchLiveStreams(vHost, httpRequest, httpResponse);
					break;
					
				case "version":
					writeSuccess(200, httpResponse, "StreamSwitchWowzaHTTPProvider v1.00: " + System.currentTimeMillis());
					break;
					
				default: 
					writeError(400, httpResponse, "Action not recognised: " + action);
					break;
			}
		} catch (MissingParameterException mpe) {
			writeError(400, httpResponse, "Missing required parameter: " + mpe.getParameterName());
		}
	}
	private void setupLiveStreams(IVHost vHost, IHTTPRequest httpRequest, IHTTPResponse httpResponse) throws MissingParameterException {
		
		String applicationName = readRequiredParameter(httpRequest, "applicationName");
		IApplication application = vHost.getApplication(applicationName);
		IApplicationInstance applicationInstance = application.getAppInstance("_definst_");
		String outputStreamName = readRequiredParameter(httpRequest, "outputStreamName");
		String inputStreamName = readRequiredParameter(httpRequest, "inputStreamName");
		
		writeLog("Starting streaming from inputStreamName: " + inputStreamName + " to outputStreamName: " + outputStreamName);
		
		Playlist inputStreamPlaylist = new Playlist(inputStreamName + "_source_playlist");	
		inputStreamPlaylist.addItem(inputStreamName, -2, -1);	
		inputStreamPlaylist.setRepeat(true);
		applicationInstance.getProperties().setProperty(inputStreamPlaylist.getName(), inputStreamPlaylist);
		
		// FIXME: filename should be included in request.
		
		Playlist pauseStreamPlaylist = new Playlist(inputStreamName + "_paused_playlist");	
		pauseStreamPlaylist.addItem("mp4:waiting.mp4", 0, -1);	
		pauseStreamPlaylist.setRepeat(true);
		applicationInstance.getProperties().setProperty(pauseStreamPlaylist.getName(), pauseStreamPlaylist);
		
		Stream outputStream = Stream.createInstance(applicationInstance, outputStreamName);
		applicationInstance.getProperties().setProperty(outputStreamName, outputStream);
		
		// Start streaming from playlist to stream.
		pauseStreamPlaylist.open(outputStream);
		
		writeSuccess(200, httpResponse, "Successfully setup live streams.");
		
	}
	
	private void switchLiveStreams(IVHost vHost, IHTTPRequest httpRequest, IHTTPResponse httpResponse) throws MissingParameterException {
		String applicationName = readRequiredParameter(httpRequest, "applicationName");
		IApplication application = vHost.getApplication(applicationName);
		IApplicationInstance applicationInstance = application.getAppInstance("_definst_");
		String outputStreamName = readRequiredParameter(httpRequest, "outputStreamName");
		String playlistName = readRequiredParameter(httpRequest, "playlistName");
		
		Stream outputStream = (Stream)applicationInstance.getProperties().getProperty(outputStreamName);
		Playlist playlist = (Playlist)applicationInstance.getProperties().getProperty(playlistName);
		playlist.open(outputStream);
		
		writeSuccess(200, httpResponse, "Successfully switched live streams.");
	}
	
	private boolean authenticate(@SuppressWarnings("unused") IHTTPRequest iHttpRequest)	{
		// TODO: Implement!
		return true;
	}
	
	private String readRequiredParameter(IHTTPRequest iHttpRequest, String parameterName) throws MissingParameterException {
		String parameterValue = iHttpRequest.getParameter(parameterName);
		if (parameterValue == null) {
			throw new MissingParameterException("Parameter: " + parameterName + " not specified.", parameterName);
		}
		return parameterValue;
	}
	
	private void writeResponse(int statusCode, IHTTPResponse resp, String response)	{
		try	{
			resp.setResponseCode(statusCode);
			OutputStream out = resp.getOutputStream();
			byte[] outBytes = response.getBytes();
			out.write(outBytes);
			writeLog("Responded with HTTP " + statusCode + ": " + response);
		}
		catch (Throwable t)	{
			resp.setResponseCode(500);
			writeLog("Responded with HTTP 500: Unknown error: " + t.getMessage());
		}
	}
	private void writeSuccess(int statusCode, IHTTPResponse resp, String response) {
		writeResponse(statusCode, resp, "[OK] " + response);
	}
	
	private void writeError(int statusCode, IHTTPResponse resp, String response) {
		writeLog("Request to StreamSwitchWowzaHTTPProvider failed: " + response);
		writeResponse(statusCode, resp, "[ERROR] " + response);
	}
	
	private void writeLog(String message) {
		WMSLoggerFactory.getLogger(null).info("[" + this.getClass().getName() + "] " + message);
	}
	
	
	private static class MissingParameterException extends Exception {
		private String parameterName;
		
		MissingParameterException(String message, String parameterName) {
			super(message);
			this.parameterName = parameterName;
		}
		String getParameterName() {
			return parameterName;
		}
	}
}

Henry,

Take a look at these 2 articles. With them you should be able to develop what you need.

How to dynamically control Stream class streams

How to loop a pre-roll until a live stream starts

Salvadore

Henry,

This kind of switching in the source of a stream can be problematic for players if codecs change substantially. This could be the problem. You set maxChunkLogCount in Application.xml /LiveStreamPacketizer Properties container so all chunks are logged. Then you can look at the codec info and packetizing log messages to see

<Property>
	<Name>maxChunkLogCount</Name>
	<Value>0</Value>
	<Type>Integer</Type>
</Property>

And for more debug info, enable httpAdapterDebugLog in the VHost.xml Pr

<Property>
 <Name>httpAdapterDebugLog</Name>
 <Value>true</Value>
 <Type>Boolean</Type>
</Property>

The idea of using the loop-until-live setup, which includes the scheduler, is just to get your base streams started. Then you could focus your code on building new playlist instances if necessary, and starting them on running streams.

You should be able to pull references to Streams created by the scheduler from the application instance properties container similar to what I see you doing. They are stored like this:

app.getAppInstance("_definst_").getProperties().setProperty(streamName, stream);

Get it in your HTTPProvider like this:

Stream stream1 = (Stream)appInstance("_definst_").getProperties().getProperty(streamName);

Richard

Henry,

You can control which streams are transcoded using this API.

Or you can use template naming to limit by first removing the fallback template (transrate.xml) in Application.xml /Transcoder /Templates, so it looks like this:

<Templates>${SourceStreamName}.xml</Templates>

If you have a template named “myStream.xml” in the /transcoder/templates folder, streams named “myStream” will transcoded using that template. You can make a template for each stream that you want to transcode; streams that do not match any template will not be transcoded.

Richard

Hi Salvadore.

Thanks for the links, I’ve had a quick scan through.

The module I wrote (listed in the original post) is based on How to switch streams using Stream class streams which did sound like the right thing to do. I guess from your links, I could have a single playlist with multiple items and cycle through them, but I understood that switching from Playlist to Playlist is the quickest way to do it.

Regardless, essentially I’m looking for a way to be able to swap intra HLS segment, certainly without a segment being generated that flickers between the previous source and the next source. Do you think that the cause of this is swapping between playlists (i.e. what I’ve done in my module), and that swapping between items in a single playlist is a solution to the problem?

Thanks,

Henry

I’ve done some more work on this today. To summarise:

  • The problem with HLS playback is unrelated to switching streams. A single stream delivered to QuickTime or iOS results in freezes, and sometimes shows frames from a previous time then frames from the current time.

  • When playing the same stream in VLC, it plays fine.

  • This only occurs when the stream source is a webcam streaming via a Flash based recorder. If the source is a webcam sending content via Wirescast, the HLS is fine.

  • This is specific to the multiple playlists setup. If I don’t use playlists, i.e. I’m viewing HLS directly from the source stream, this works regardless of the input type.

I tried using a single Playlist with multiple items as you suggested, and this seems to work ok. However, this doesn’t really implement what we want. We’d like the ability to dynamically set up these streams, loop some but not all streams (e.g. the ‘pause’ stream which can occur in the middle of an event rather than pre/post roll), and we’d like to ability so switch to a named stream rather than just swapping to the next item in a playlist. As such, the multi-playlist solution seemed to tick all the boxes.

As an example of the problem, this is a ‘broken’ TS segment as produced by Wowza in the multiple playlists setup: http://sandbox.kuluvalley.com/docs/wowza.ts If you view this in QuickTime or iOS you’ll see the video pauses from time to time. In VLC, it plays fine.

So, do you have any idea why Wowza would be producing ‘bad’ content in this case? If not, any suggestions as to what else I could do to diagnose the issue.

Thanks,

Henry

Hi Richard.

Thanks for the detailed response. I’ve spent some time setting up an improved testing rig (hence the delay in getting back to you.) Previously I was running Wowza in a linux VM on my (now last gen) Mac Pro, along with our app and Wirecast. The load was fairly high on the machine, so I had a suspicion this was leading to transcoding issues. I’ve now moved Wowza off onto a separate, reasonably spec’d (core i7, 8GB RAM) machine running Linux. This seems to have fixed the stuttering issue I was getting in HLS, although I can really explain why I was getting this only on iOS and not VLC.

I’ve now got two issues switching streams, specific to HLS. RTMP streams seem to work great.

i) Occasionally playback dies on iOS with a ‘Could not load movie’ error. I’ve written a simple curl script to poll the M3U8 during switch over – it turns out Wowza returns a 404 for a second or two around this time. This causes the iOS error. I can work around this, since we’ll be proxying the M3U8, with some caching, but regardless, are you aware of this issue? Is there a workaround in Wowza to prevent this?

ii) During switchover, there is a pause in the HLS stream. I had been assuming (hoping!) that if the switchover occured mid way through a segment generation, the segment would be generated with part of stream 1 and part of stream 2. However, by doing lots of waving at oppurtune times at the two test cameras I have, I believe the following is happening:

i) Output is from camera 1. HLS segment is mid way through generation.

ii) Request is made to switch over to camera 2.

iii) Segment generation from camera 1 is stopped, and segment is written.

iv) Wowza pauses until a new segment is started for camera 2 (presumably it was already generating segments for camera 2, so it waits for the current segment to complete.)

v) Subsequent segments from camera 2 are returned.

This leads to a pause of (presumably) up to a segment, and thus missed content from camera 2.

Is this correct? If so, is there some way to ‘fix’ this? The switchover seems to be perfect for RTMP, I had hoped for similar behaviour in HLS – although I appreciate it may be hard to change segment generation mid way through. I would imagine I can minimise the pause / content loss by reducing segment size, but I had hoped for a better solution than this.

FWIW, I’m viewing transcoded output – I’d hoped that by using the transcoder this would get over issues with different input sizes, bit rates. And this is the log output during a stream switch (note the ‘Successfully switched live stream’ message mid way through):

2014-02-25	12:40:57	GMT	comment	server	INFO	200	-	LiveStreamPacketizerSmoothStreaming.addFragment[livedesktop/_definst_/WVkgzwogYkm]: Add chunk: type:video id:3320 count:15 duration:1063	-	-	-	4225.023	-	-	-
2014-02-25	12:40:57	GMT	comment	server	INFO	200	-	LiveStreamPacketizerCupertino.endChunkTS[livedesktop/_definst_/WVkgzwogYkm_360p]: Add chunk: id:422 mode:TS[H264,AAC] a/v/k:42/120/2 duration:7926	-	-	-	4225.1	-	-	-
2014-02-25	12:40:57	GMT	comment	server	INFO	200	-	LiveStreamPacketizerSanJose.endChunkTS[livedesktop/_definst_/WVkgzwogYkm_360p]: Add chunk: id:422 a/v/k:126/121/2 duration:7926	-	-	-	4225.1	-	-	-	-	-	-
2014-02-25	12:40:57	GMT	comment	server	INFO	200	-	LiveStreamPacketizerSmoothStreaming.addFragment[livedesktop/_definst_/WVkgzwogYkm_360p]: Add chunk: type:video id:896 count:60 duration:4100	-	-	-	4225.1	-	-	-	-
2014-02-25	12:40:57	GMT	comment	server	INFO	200	-	LiveStreamPacketizerSanJose.endChunkTS[livedesktop/_definst_/stream1HMRm_encoder_360p]: Add chunk: id:282 a/v/k:290/61/1 duration:6700	-	-	-	4225.506	-	-	-	-
2014-02-25	12:40:57	GMT	comment	server	INFO	200	-	LiveStreamPacketizerSmoothStreaming.addFragment[livedesktop/_definst_/stream1HMRm_encoder_360p]: Add chunk: type:audio id:1393 count:87 duration:2020	-	-	-	4225.507	-	-
2014-02-25	12:40:57	GMT	comment	server	INFO	200	-	LiveStreamPacketizerCupertino.endChunkTS[livedesktop/_definst_/stream1HMRm_encoder_360p]: Add chunk: id:282 mode:TS[H264,AAC] a/v/k:96/60/1 duration:6700	-	-	-	4225.506	-
2014-02-25	12:40:57	GMT	comment	server	INFO	200	-	LiveStreamPacketizerSmoothStreaming.addFragment[livedesktop/_definst_/stream1HMRm_encoder_360p]: Add chunk: type:video id:390 count:60 duration:6633	-	-	-	4225.507	-	-
2014-02-25	12:40:57	GMT	comment	server	INFO	200	-	LiveStreamPacketizerSmoothStreaming.addFragment[livedesktop/_definst_/stream1HMRm_360p]: Add chunk: type:video id:1035 count:60 duration:3990	-	-	-	4225.594	-	-	-
2014-02-25	12:40:57	GMT	comment	server	INFO	200	-	HTTPStreamerAdapterSmoothStreamer.canHandle[livedesktop/mp4:WVkgzwogYkm_360p/chunklist_w61674231.m3u8]: false	-	-	-	4225.744	-	-	-	-	-	-	-
2014-02-25	12:40:57	GMT	comment	server	INFO	200	-	HTTPStreamerAdapterCupertinoStreamer.onPlaylist: livedesktop/mp4:WVkgzwogYkm_360p/chunklist_w61674231.m3u8	-	-	-	4225.745	-	-	-	-	-	-	-
2014-02-25	12:40:58	GMT	comment	server	INFO	200	-	HTTPStreamerAdapterSmoothStreamer.canHandle[stream-control]: false	-	-	-	4225.973	-	-	-	-	-	-	-	-	-	-	-	-
2014-02-25	12:40:58	GMT	comment	server	INFO	200	-	HTTPStreamerAdapterSanJoseStreamer.canHandle[stream-control]: false	-	-	-	4225.974	-	-	-	-	-	-	-	-	-	-	-	-
2014-02-25	12:40:58	GMT	comment	server	INFO	200	-	HTTPStreamerAdapterDvrChunkStreamer.canHandle[stream-control]: false	-	-	-	4225.974	-	-	-	-	-	-	-	-	-	-	-	-
2014-02-25	12:40:58	GMT	comment	server	INFO	200	-	[com.kuluvalley.wms.http.StreamSwitchWowzaHTTPProvider] Responded with HTTP 200: [OK] Successfully switched live stream for outputStreamName: WVkgzwogYkm to playlistName: stream1HMRm_encoder_playlist	-	-	-	4225.976	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
2014-02-25	12:40:58	GMT	comment	server	INFO	200	-	LiveStreamPacketizerSmoothStreaming.addFragment[livedesktop/_definst_/stream1HMRm]: Add chunk: type:video id:4197 count:15 duration:987	-	-	-	4225.981	-	-	-	-
2014-02-25	12:40:58	GMT	comment	server	INFO	200	-	Stream.switch[livedesktop/_definst_/WVkgzwogYkm]: index: 0 name:stream1HMRm_encoder start:-2 length:-1	-	-	-	4226.001	-	-	-	-	-	-	-	-
2014-02-25	12:40:58	GMT	comment	server	INFO	200	-	TranscodingSession.resetStream[livedesktop/_definst_/WVkgzwogYkm]	-	-	-	4226.002	-	-	-	-	-	-	-	-	-	-	-	-
2014-02-25	12:40:58	GMT	comment	server	INFO	200	-	LiveStreamPacketizerCupertino.endChunkTS[livedesktop/_definst_/WVkgzwogYkm]: Add chunk: id:434 mode:TS[H264,SPEEX] a/v/k:0/30/2 duration:1949	-	-	-	4226.002	-	-	-
2014-02-25	12:40:58	GMT	comment	server	INFO	200	-	LiveStreamPacketizerSmoothStreaming.addFragment[livedesktop/_definst_/WVkgzwogYkm]: Add chunk: type:video id:3321 count:15 duration:987	-	-	-	4226.003	-	-	-	-
2014-02-25	12:40:58	GMT	comment	server	INFO	200	-	LiveStreamPacketizerSanJose.endChunkTS[livedesktop/_definst_/WVkgzwogYkm]: Add chunk: id:433 a/v/k:98/31/2 duration:1949	-	-	-	4226.002	-	-	-	-	-
2014-02-25	12:40:58	GMT	comment	server	INFO	200	-	LiveStreamPacketizerSmoothStreaming.resetStream[livedesktop/_definst_/WVkgzwogYkm_360p]	-	-	-	4226.003	-	-	-	-	-	-	-	-	-	-
2014-02-25	12:40:58	GMT	comment	server	INFO	200	-	LiveStreamPacketizerCupertino.resetStream[livedesktop/_definst_/WVkgzwogYkm_360p]	-	-	-	4226.003	-	-	-	-	-	-	-	-	-	-
2014-02-25	12:40:58	GMT	comment	server	INFO	200	-	LiveStreamPacketizerSmoothStreaming.resetStream[livedesktop/_definst_/WVkgzwogYkm]	-	-	-	4226.003	-	-	-	-	-	-	-	-	-	-
2014-02-25	12:40:58	GMT	comment	server	INFO	200	-	LiveStreamPacketizerSanJose.resetStream[livedesktop/_definst_/WVkgzwogYkm_360p]	-	-	-	4226.003	-	-	-	-	-	-	-	-	-	-	-
2014-02-25	12:40:58	GMT	comment	server	INFO	200	-	LiveStreamPacketizerSanJose.resetStream[livedesktop/_definst_/WVkgzwogYkm]	-	-	-	4226.006	-	-	-	-	-	-	-	-	-	-	-
2014-02-25	12:40:58	GMT	comment	server	INFO	200	-	LiveStreamPacketizerCupertino.resetStream[livedesktop/_definst_/WVkgzwogYkm]	-	-	-	4226.003	-	-	-	-	-	-	-	-	-	-	-
2014-02-25	12:40:58	GMT	comment	server	INFO	200	-	LiveStreamPacketizerSanJose.handlePacket[livedesktop/_definst_/WVkgzwogYkm]: Video codec: H264	-	-	-	4226.007	-	-	-	-	-	-	-	-	-
2014-02-25	12:40:58	GMT	comment	server	INFO	200	-	LiveStreamPacketizerCupertino.handlePacket[livedesktop/_definst_/WVkgzwogYkm]: Video codec:H264 isCompatible:true	-	-	-	4226.007	-	-	-	-	-	-

Thanks,

Henry

Actually, I think I may have a fix for problem ii). The second camera source was providing key frames every 6-7 seconds. I made an ‘educated’ guess that the segment generator needed a source keyframe to start afresh, so by changing the source to have a keyframe interval of 1s suddenly transitions are pretty snappy. I’ll play around with this some more, but hopefully that’s done it.

If you do, however, have any thoughts as to why Wowza would be returning a 404 during switch over, that’d be much appreciated.

Thanks,

Henry

Sorry for the spam, but I’ve had another thought on this which may be a better solution – if it’s possible (!) A problem we have with the current setup is that all the streams are coming into the transcoder enabled, live endpoint, so all the streams are being transcoded. In reality, we only care about the switched output stream being transcoded; if we could achieve this it would save us a load of CPU resource (and transcoder licenses :wink: )

So, is something like this possible? Two (Wowza) applications are setup:

LiveInput – accepts n input streams and uses my plugin to generate a single, RTMP output stream. No transcoding is enabled.

LiveOutput – accepts the RTMP input from LiveInput, generates HLS streams (etc) using the transcoder.

Given that the switching for RTMP seems to work regardless of source keyframes, this would also get around fixed length GOP requirements for source streams.

Thanks,

Henry

Thanks for the info Richard.

In the end we got our input / output solution working. This seemed to tick all our requirement boxes:

  1. We can have arbitrary stream names rather than having to adhere to a transcoder template naming convention.

  2. We only transcode the output stream.

  3. It gets around the problem with Wowza 404-ing in between stream switches for HLS since HLS is given a consistent RTMP stream and is no longer aware of switching.

  4. It gets around a disparity between the RTMP output and HLS output. I never quite figured this out, but post switch, HLS could pause for longer than RTMP. More regular source key frames seemed to improve, but not solve this. Anyway, given that we’re producing an HLS stream from the switched RTMP output, this is no longer an issue.

If anyone else decides to adopt a similar approach, the trick (if you can call it such a thing) is for the output application to start consuming RTMP using the following code:

outputApplicationInstance.startMediaCasterStream(inputStreamName, "liverepeater");

inputStreamName can be a .stream file name or in my case I use an alias which is resolved to the input stream address (rtmp://localhost/liveinput/muxedinputstreamname, say) using an IMediaStreamNameAliasProvider

Anyway, thanks again for the assistance.

Henry