Sending a large response from an HTTP Provider

I’ve created a Wowza HttpProvider (base class HTTProvider2Base) which returns all the access logs for a given month, as a single gzipped response. It works great for smaller amounts of data (up to about 110MB uncompressed, about 6MB compressed) but fails when the response gets larger.

It seems like Wowza Server may be trying to buffer the entire response before sending it to the client, even though I am explicitly flushing the output stream as I write to it. The server takes a long time to respond, and when it finally does it just sends a small amount of data and closes the connection:

curl http://localhost:8086/accesslogs/2012-04 -o logs.txt.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0 24.7M    0 36768    0     0   2182      0  3:17:53  0:00:16  3:17:37  9737
curl: (18) transfer closed with 25870859 bytes remaining to read

Here’s the Java code I use to read the log files and write them to the output stream. I’ve removed the error handling and logging code for clarity:

GZIPOutputStream out = new GZIPOutputStream(resp.getOutputStream(), 4096);
resp.setResponseCode(200);
resp.setHeader("Content-Encoding", "gzip");
File[] files = getLogFiles(requestedYear, requestedMonth);
byte[] buffer = new byte[4096];
for (File file : files) {
    InputStream input = new BufferedInputStream(
        new FileInputStream(file.getAbsolutePath()));
    int bytesRead = 0;
    while ((bytesRead = input.read(buffer)) > 0){
        out.write(buffer, 0, bytesRead);
        out.flush();
    }
    input.close();
}
out.close();

Any idea why the response is getting buffered? Is there a Wowza server setting I need to change? Is this a limitation of Wowza HttpProviders? All the examples of HttpProviders I’ve found have small responses and build the response in a string or something before sending to the output stream, so maybe HttpProviders aren’t designed to handle larger responses.

I’ve tried the following to no effect:

  • adding a call to out.finish()

  • changing the size of the buffer

  • changing the size of the internal buffer for GZIPOutputStream

  • not using gzip at all, just sending the logs as plain text

    I’m using Wowza Media Server 2.2.4 on Windows 7, and I see the same behaviour on our hosted Fedora 8/Wowza 2.1.2 instance in EC2.

    Thanks!

I’ve done some more testing, with and without gzip.

  • With gzip, when the content-length is around 1MB (1048576 bytes), it fails to send the entire response.

  • Without gzip, when the content-length is around 100MB (104857600 bytes), it fails to send the entire response.

    Based on the response sizes being exactly 1MB or 100MB, it seems pretty obvious to me that I’m hitting some buffer limits somewhere.

    I added logging on the server that shows the data being written to the output stream. The server logs don’t show any errors, and I’m not getting any exceptions.

    The request flow goes like this:

  • client connects and sends http request with headers, including accept-encoding: gzip

  • server reads the log file for the requested date, and writes it to the gzip output stream

  • when the reading/writing is complete, server sends http headers to client, including content-encoding, and content-length

  • server sends 33540 bytes of content to the client

  • connection gets closed (presumably by the server)

    here’s the transcript of what happens with curl:

    curl --compressed -v --digest -u username:password http://<ip>:8086/accesslogs/20120421 -o wowza_access.log.2012-04-21
    * About to connect() to <ip> port 8086 (#0)
    * Trying <ip>...
      % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                     Dload  Upload   Total   Spent    Left  Speed
      0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
    * connected
    * Connected to <ip> (<ip>) port 8086 (#0)
    * Server auth using Digest with user 'admin'
    > GET /accesslogs/20120421 HTTP/1.1
    > User-Agent: curl/7.24.0 (i386-pc-win32) libcurl/7.24.0 OpenSSL/0.9.8t zlib/1.2.5
    > Host: <ip>:8086
    > Accept: */*
    > Accept-Encoding: deflate, gzip
    >
      0     0    0     0    0     0      0      0 --:--:--  0:00:07 --:--:--     0
    < HTTP/1.1 200 OK
    < Content-Encoding: gzip
    < Content-Type: text/plain
    < Connection: Keep-Alive
    < Server: FlashCom/3.5.2
    < Cache-Control: no-cache
    < Content-Length: 1430462
    <
    { [data not shown]
    * transfer closed with 1396922 bytes remaining to read
      2 1396k    2 33540    0     0   4516      0  0:05:16  0:00:07  0:05:09  7733
    * Closing connection #0
    curl: (18) transfer closed with 1396922 bytes remaining to read
    
    

This has never worked well. For uploading/downloading files, it’s easier to just install a web server.

Richard

But do you know what make that fails? We are frustred too because we are working in a stats system and after a lot of work done this issue will need a complete change in our work. You must put some advice in some place of your site about that problem to prevent other people to try to make thinhs that is not possible.

I think we have same problem.

We are sending jpeg images and around ~35K is sent and then conection is closed.

Strange.

Wowza 3.0.4

Here’s the entire source code for my http provider, in it’s current state. To install it, compile it and put the .jar into the Wowza lib folder. Then add the following into VHost.xml, in the admin HostPort.HTTPProviders section:

<HTTPProvider>
	<BaseClass>tv.epresence.wms.AccessLogProvider</BaseClass>
	<RequestFilters>accesslogs*</RequestFilters>
	<AuthenticationMethod>admin-digest</AuthenticationMethod>
</HTTPProvider>

Add the path to the logs folder in the VHost.Properties section:

<Property>
	<Name>rootLogPath</Name>
	<Value>/usr/local/WowzaMediaServer/logs</Value>
</Property>

AccessLogProvider.java:

package tv.epresence.wms;
import java.io.*;
import java.util.*;
import java.util.zip.*;
import java.text.*;
import java.util.regex.*;
import com.wowza.wms.vhost.*;
import com.wowza.wms.http.*;
import com.wowza.wms.logging.*;
public class AccessLogProvider extends HTTProvider2Base {
	
	public static WMSLogger log;
	static {
		log = WMSLoggerFactory.getLogger(null);
	}
	
	private AccessLogRequest parseRequestParameters(IHTTPRequest req) {
		Date startDate;
		Date endDate;
		
		String yearParam   = req.getParameter("year");
		String monthParam  = req.getParameter("month");
		String dayParam    = req.getParameter("day");
		String startParam  = req.getParameter("start");
		String endParam    = req.getParameter("end");
		String periodParam = req.getParameter("period");
		
		int requestedYear;
		int requestedMonth;
		int requestedDay;
		
		try {
			if (yearParam != null && monthParam != null) {
				
				// Use year, month parameters if available
				
				requestedYear  = Integer.parseInt(yearParam);
				requestedMonth = Integer.parseInt(monthParam);
				
				Calendar cal = Calendar.getInstance();
				if (dayParam != null) {
					requestedDay = Integer.parseInt(dayParam);
					cal.set(requestedYear, requestedMonth - 1, requestedDay, 0, 0, 0);
					startDate = cal.getTime();
					
					cal.add(Calendar.DATE, 1);
					endDate = cal.getTime();
				}
				else {
					cal.set(requestedYear, requestedMonth - 1, 1, 0, 0, 0);
					startDate = cal.getTime();
					
					cal.add(Calendar.MONTH, 1);
					endDate = cal.getTime();
				}
			}
			else if (startParam != null && endParam != null) {
				
				// Use start, end parameters if available
				
				DateFormat df = new SimpleDateFormat("yyyyMMdd");
				startDate = df.parse(startParam); 
				endDate   = df.parse(endParam); 
			}
			else {
				if (periodParam == null) {
					// Match URL /accesslogs/yearmonth(day) (eg. 201204, 20120401)
					String uri = req.getRequestURI();
					periodParam = uri.substring(uri.lastIndexOf('/') + 1);
				}
				
				Calendar cal = Calendar.getInstance();
				
				if (periodParam.length() == 8) {
					DateFormat df = new SimpleDateFormat("yyyyMMdd");
					startDate = df.parse(periodParam); 
					cal.setTime(startDate);
					cal.add(Calendar.DATE, 1);
					endDate = cal.getTime();
				}
				else if (periodParam.length() == 6) {
					DateFormat df = new SimpleDateFormat("yyyyMM");
					startDate = df.parse(periodParam); 
					cal.setTime(startDate);
					cal.add(Calendar.MONTH, 1);
					endDate = cal.getTime();
				}
				else {
					throw new Exception("AccessLogProvider: invalid period specified: " + periodParam);
				}
			}
		}
		catch (Exception e) {
			log.error(e.toString());
			return null;
		}
		return new AccessLogRequest(startDate, endDate); 
	}
	
	private File[] getRequestedFiles(IVHost vhost, AccessLogRequest request) {
		List<File> files = new ArrayList<File>();
		
		String rootPath = vhost.getProperties().getProperty("rootLogPath").toString();
		rootPath = rootPath.replace('/', File.separatorChar);
		
		File dir = new File(rootPath);
		
		if (!dir.isDirectory()) {
			log.error("AccessLogProvider: rootLogPath is not a directory: " + rootPath);
			return files.toArray(new File[files.size()]);
		}
		
		Pattern pattern = Pattern.compile(".*access\\.log\\.(\\d{4})-(\\d{2})-(\\d{2})");
		for (File file : dir.listFiles()) {
			if (file.getName() == "." || file.getName() == "..") {
				continue;
			}
			Matcher m = pattern.matcher(file.getName());
			if (m.matches()) {
				int year  = Integer.parseInt(m.group(1));
				int month = Integer.parseInt(m.group(2));
				int day   = Integer.parseInt(m.group(3));
				
				Date fileDate = MakeDate(year, month, day);
				if (DateBetween(fileDate, request.startDate, request.endDate)) {
					files.add(file);
				}
			}
		}
		return files.toArray(new File[files.size()]);
	}
	public boolean DateBetween(Date date, Date start, Date end) {
		return (date.after(start) && date.before(end));
	}
	
	public Date MakeDate(int year, int month, int day) {
		Calendar cal = Calendar.getInstance();
		cal.set(year, month - 1, day, 0, 0, 0);
		return cal.getTime();
	}
	
	public void onHTTPRequest(IVHost vhost, IHTTPRequest httpRequest, IHTTPResponse resp)
	{
		if (!doHTTPAuthentication(vhost, httpRequest, resp)) {
			return;
		}
		
		String encodingHeader = httpRequest.getHeader("Accept-Encoding");
		 
		boolean gzip = (encodingHeader != null) && encodingHeader.contains("gzip");
		OutputStream out;
		GZIPOutputStream gzipOut = null;
		try {
			if (gzip) {
				out = gzipOut = new GZIPOutputStream(resp.getOutputStream(), 4096);
			}
			else {
				out = new BufferedOutputStream(resp.getOutputStream());
			}
		} catch (IOException e) {
			log.error("AccessLogProvider: Exception: " + e.toString());
			resp.setResponseCode(500);
			return;
		}
		try {
			AccessLogRequest request = parseRequestParameters(httpRequest);
			if (request != null) {
				resp.setResponseCode(200);
				resp.setHeader("Content-Type", "text/plain");
				
				if (gzip) {
					resp.setHeader("Content-Encoding", "gzip");
				}
				
				File[] files = getRequestedFiles(vhost, request);
				byte[] buffer = new byte[4096];
				
				DateFormat df = new SimpleDateFormat("yyyyMMdd");
				log.info("AccessLogProvider: start=" + df.format(request.startDate) + ", end=" + df.format(request.endDate) + " files=" + files.length + (gzip ? " (gzip)" : ""));
				
				for (File file : files) {
					log.info("AccessLogProvider: sending " + file.getName());
					InputStream input = new BufferedInputStream(new FileInputStream(file.getAbsolutePath()));
					
					int bytesRead = 0;
					int totalBytesRead = 0;
					while((bytesRead = input.read(buffer)) > 0){
						totalBytesRead += bytesRead;
						log.info(String.format("AccessLogProvider: %1$d bytes read", totalBytesRead));
						try {
							out.write(buffer, 0, bytesRead);
							out.flush();
							log.info(String.format("AccessLogProvider: %1$d bytes written", totalBytesRead));
						}
						catch (IOException e) {
							log.error("AccessLogProvider: write failed: " + e.toString());
							break;
						}
					}
					input.close();
				}
				out.flush();
				
				if (gzip) {
					gzipOut.flush();
					gzipOut.finish();
					gzipOut.close();
				}
				
				out.close();
			}
			else {
				log.error("AccessLogProvider: invalid URI: " + httpRequest.getRequestURI());
				resp.setResponseCode(404);
			}
		}
		catch (Exception e) {
			log.error("AccessLogProvider: Exception: " + e.toString());
			resp.setResponseCode(500);
		}
	}
}

I met the same problem.

I send mp3/mp4 and around 2.7M is sent and then the response failed. Wowza 3.1.2.

Anyone can help?