Hi there,
We have been using multiple Wowza Streaming Engine instances in production for years now, and we have identified some problems in our production environment that I would like some help getting to the bottom of.
We have one node whose job it is to stream VOD content from Amazon S3 to the end user. We are using the built in Wowza Media cache with S3 Source to pull videos into the cache. Once videos are in the media cache, switching between them as an end user is very fast, and load time is always consistent. Unfortunately, videos that need to be pulled from S3 “fresh” have inconsistent load times. As an end user with 100 mbps download speed, some videos load in less than one second, and others take up to ten seconds. These videos are between 1 and 4 megabytes. We believe this is due to interactions between S3 and the media cache, but we are not entirely positive.
We see in the wowzastreamingengine_access.log that the step between receiving the request to play and the log that says the stream is playing in the form: "2016-11-09 21:53:58 UTC play stream INFO 200 " is where the delay happens. We believe it’s possible that requests to S3 are getting backed up. We have up to 150 users at once loading new media from only 4 different S3 buckets, so we believe we may be getting choked somewhere in the pipeline.
Is there anything in the Wowza performance tuning we can tweak to prevent these delays? Inside Media Cache Tuning, I was thinking of increasing all the settings in Performance Tuning > Media Cache Tuning from the automatic defaults. Since this node only services VOD from S3, it can probably take the extra threads and bandwidth.
I was also wondering if we are possibly exhausting the http connection pool for the s3 object that fetches media. In my own independent Java AWS testing, I noticed that the default max HTTP connections for a single AmazonS3 object is only 50. If these connections are not well managed the pool can be exhausted, leading to hang ups. I am not sure how Wowza’s implementation of Media Cache with S3 source handles http connections, but is it possible that these are getting used up?
I am happy to provide logs, xml config files, and specifications of our cloud servers to help debug our issues.
Thanks!