Live stream performance at 1gbps

We had two rather big live streams going out yesterday (Dalai Lama on Wisoconsin Public TV’s site, 2 hours in the morning, 90 mins in the afternoon), and from what I saw, I think our network configuration kept the traffic from exceeding 1gbps. We are still looking for the root of the problem, as the system is on a 10 gig interface with more than 10gig upstream. We didn’t see any striking server errors in logs, but when traffic went from 900 streams to 1400+ streams without our outgoing bandwidth changing, it became obvious we had a bottleneck at 1gbps. Streams were downshifting to lower qualities and everyone got to see the event, but costantly switching Flash players weren’t really that much fun to watch (for those of us who knew what was happening).

I’m just throwing out a few things here, hoping to get some feedback where to possibly improve things in the future.

a few data points up front:

Server is a Dell r720 or something like that, 3 years old with 2 Xeon E5530 2.4ghz CPUs (8 cores, 16 threads) 24GB RAM, four 1 gbps interfraces and one 10gbps Intel X-520-2 10 gig card (which is the only interface Wowza is using, and it uses it exclusively) OS is 2008 r2 64-bit Enterprise edition.

running wowza 3.5.0 and two transcoder sessions

vhost file content

<Root>
	<VHost>
		<HostPortList>
			<HostPort>
				<ProcessorCount>16</ProcessorCount>
				<IpAddress>128.nnn.nnn.nnn</IpAddress>
				<!-- 128.nnn.nnn.nnn is our fixed 10 gig number -->
				<!-- Separate multiple ports with commas -->
				<!-- 80: HTTP, RTMPT -->
				<!-- 554: RTSP -->
				<Port>1935,80,443,554</Port>
				<SocketConfiguration>
					<ReuseAddress>true</ReuseAddress>
					<!-- suggested settings for video on demand applications -->
					<ReceiveBufferSize>65000</ReceiveBufferSize>
					<ReadBufferSize>65000</ReadBufferSize>
					<SendBufferSize>65000</SendBufferSize>
					<!-- suggest settings for low latency chat and video recording applications
					<ReceiveBufferSize>32000</ReceiveBufferSize>
					<ReadBufferSize>32000</ReadBufferSize>
					<SendBufferSize>32000</SendBufferSize>
					-->
					<KeepAlive>true</KeepAlive>
					<!-- <TrafficClass>0</TrafficClass> -->
					<!-- <OobInline>false</OobInline> -->
					<!-- <SoLingerTime>-1</SoLingerTime> -->
					<!-- <TcpNoDelay>false</TcpNoDelay> -->
					<AcceptorBackLog>100</AcceptorBackLog>
				</SocketConfiguration>
				<HTTPStreamerAdapterIDs>cupertinostreaming,smoothstreaming,sanjosestreaming,dvrchunkstreaming</HTTPStreamerAdapterIDs>
				<HTTPProviders>
					<HTTPProvider>
						<BaseClass>com.wowza.wms.http.HTTPCrossdomain</BaseClass>
						<RequestFilters>*crossdomain.xml</RequestFilters>
						<AuthenticationMethod>none</AuthenticationMethod>
					</HTTPProvider>
					<HTTPProvider>
						<BaseClass>com.wowza.wms.http.HTTPClientAccessPolicy</BaseClass>
						<RequestFilters>*clientaccesspolicy.xml</RequestFilters>
						<AuthenticationMethod>none</AuthenticationMethod>
					</HTTPProvider>
					<HTTPProvider>
						<BaseClass>com.wowza.wms.http.HTTPProviderMediaList</BaseClass>
						<RequestFilters>*jwplayer.rss|*jwplayer.smil|*medialist.smil|*manifest-rtmp.f4m</RequestFilters>
						<AuthenticationMethod>none</AuthenticationMethod>
					</HTTPProvider>
					<HTTPProvider>
						<BaseClass>com.wowza.wms.http.HTTPServerVersion</BaseClass>
						<RequestFilters>*</RequestFilters>
						<AuthenticationMethod>none</AuthenticationMethod>
					</HTTPProvider>
				</HTTPProviders>
			</HostPort>
			
			<!-- 443 with SSL -->
			<!--
			<HostPort>
				<ProcessorCount>4</ProcessorCount>
				<IpAddress>*</IpAddress>
				<Port>443</Port>
				<SSLConfig>
					<KeyStorePath>${com.wowza.wms.context.VHostConfigHome}/conf/keystore.jks</KeyStorePath>
					<KeyStorePassword>[password]</KeyStorePassword>
					<KeyStoreType>JKS</KeyStoreType>
					<SSLProtocol>TLS</SSLProtocol>
					<Algorithm>SunX509</Algorithm>
					<CipherSuites></CipherSuites>
					<Protocols></Protocols>
				</SSLConfig>
				<SocketConfiguration>
					<ReuseAddress>true</ReuseAddress>
					<ReceiveBufferSize>65000</ReceiveBufferSize>
					<ReadBufferSize>65000</ReadBufferSize>
					<SendBufferSize>65000</SendBufferSize>
					<KeepAlive>true</KeepAlive>
					<AcceptorBackLog>100</AcceptorBackLog>
				</SocketConfiguration>
				<HTTPStreamerAdapterIDs>cupertinostreaming,smoothstreaming,sanjosestreaming,dvrchunkstreaming</HTTPStreamerAdapterIDs>
				<HTTPProviders>
					<HTTPProvider>
						<BaseClass>com.wowza.wms.http.HTTPCrossdomain</BaseClass>
						<RequestFilters>*crossdomain.xml</RequestFilters>
						<AuthenticationMethod>none</AuthenticationMethod>
					</HTTPProvider>
					<HTTPProvider>
						<BaseClass>com.wowza.wms.http.HTTPClientAccessPolicy</BaseClass>
						<RequestFilters>*clientaccesspolicy.xml</RequestFilters>
						<AuthenticationMethod>none</AuthenticationMethod>
					</HTTPProvider>
					<HTTPProvider>
						<BaseClass>com.wowza.wms.http.HTTPProviderMediaList</BaseClass>
						<RequestFilters>*jwplayer.rss|*jwplayer.smil|*medialist.smil|*manifest-rtmp.f4m</RequestFilters>
						<AuthenticationMethod>none</AuthenticationMethod>
					</HTTPProvider>
					<HTTPProvider>
						<BaseClass>com.wowza.wms.http.HTTPServerVersion</BaseClass>
						<RequestFilters>*</RequestFilters>
						<AuthenticationMethod>none</AuthenticationMethod>
					</HTTPProvider>
				</HTTPProviders>
			</HostPort>
			-->
			
			<!-- Admin HostPort -->
			<HostPort>
				<ProcessorCount>16</ProcessorCount>
				<IpAddress>128.nnn.nnn.nnn</IpAddress>
				<Port>8086</Port>
				<SocketConfiguration>
					<ReuseAddress>true</ReuseAddress>
					<ReceiveBufferSize>16000</ReceiveBufferSize>
					<ReadBufferSize>16000</ReadBufferSize>
					<SendBufferSize>16000</SendBufferSize>
					<KeepAlive>true</KeepAlive>
					<AcceptorBackLog>100</AcceptorBackLog>
				</SocketConfiguration>
				<HTTPStreamerAdapterIDs></HTTPStreamerAdapterIDs>
				<HTTPProviders>
					<HTTPProvider>
						<BaseClass>com.wowza.wms.http.streammanager.HTTPStreamManager</BaseClass>
						<RequestFilters>streammanager*</RequestFilters>
						<AuthenticationMethod>admin-digest</AuthenticationMethod>
					</HTTPProvider>
					<HTTPProvider>
						<BaseClass>com.wowza.wms.http.HTTPServerInfoXML</BaseClass>
						<RequestFilters>serverinfo*</RequestFilters>
						<AuthenticationMethod>admin-digest</AuthenticationMethod>
					</HTTPProvider>
					<HTTPProvider>
						<BaseClass>com.wowza.wms.http.HTTPConnectionInfo</BaseClass>
						<RequestFilters>connectioninfo*</RequestFilters>
						<AuthenticationMethod>admin-digest</AuthenticationMethod>
					</HTTPProvider>
					<HTTPProvider>
						<BaseClass>com.wowza.wms.http.HTTPConnectionCountsXML</BaseClass>
						<RequestFilters>connectioncounts*</RequestFilters>
						<AuthenticationMethod>admin-digest</AuthenticationMethod>
					</HTTPProvider>
					<HTTPProvider>
						<BaseClass>com.wowza.wms.transcoder.httpprovider.HTTPTranscoderThumbnail</BaseClass>
						<RequestFilters>transcoderthumbnail*</RequestFilters>
						<AuthenticationMethod>admin-digest</AuthenticationMethod>
					</HTTPProvider>
					<HTTPProvider>
						<BaseClass>com.wowza.wms.http.HTTPProviderMediaList</BaseClass>
						<RequestFilters>medialist*</RequestFilters>
						<AuthenticationMethod>admin-digest</AuthenticationMethod>
					</HTTPProvider>
					<HTTPProvider>
						<BaseClass>com.wowza.wms.livestreamrecord.http.HTTPLiveStreamRecord</BaseClass>
						<RequestFilters>livestreamrecord*</RequestFilters>
						<AuthenticationMethod>admin-digest</AuthenticationMethod>
					</HTTPProvider>
					<HTTPProvider>
						<BaseClass>com.wowza.wms.http.HTTPServerVersion</BaseClass>
						<RequestFilters>*</RequestFilters>
						<AuthenticationMethod>none</AuthenticationMethod>
					</HTTPProvider>
				</HTTPProviders>
			</HostPort>
		
		</HostPortList>
		
		<HTTPStreamerAdapters>
			<HTTPStreamerAdapter>
				<ID>smoothstreaming</ID>
				<Name>smoothstreaming</Name>
				<Properties>
				</Properties>
			</HTTPStreamerAdapter>
			<HTTPStreamerAdapter>
				<ID>cupertinostreaming</ID>
				<Name>cupertinostreaming</Name>
				<Properties>
				</Properties>
			</HTTPStreamerAdapter>
			<HTTPStreamerAdapter>
				<ID>sanjosestreaming</ID>
				<Name>sanjosestreaming</Name>
				<Properties>
				</Properties>
			</HTTPStreamerAdapter>
			<HTTPStreamerAdapter>
				<ID>dvrchunkstreaming</ID>
				<Name>dvrchunkstreaming</Name>
				<Properties>
				</Properties>
			</HTTPStreamerAdapter>
		</HTTPStreamerAdapters>
		<HandlerThreadPool>
			<PoolSize>120</PoolSize>
		</HandlerThreadPool>
		<TransportThreadPool>
			<PoolSize>80</PoolSize>
		</TransportThreadPool>
		<IdleWorkers>
			<WorkerCount>4</WorkerCount>
			<CheckFrequency>50</CheckFrequency>
			<MinimumWaitTime>5</MinimumWaitTime>
		</IdleWorkers>
		<NetConnections>
			<ProcessorCount>16</ProcessorCount>
			<IdleFrequency>250</IdleFrequency>
			<SocketConfiguration>
				<ReuseAddress>true</ReuseAddress>
				<ReceiveBufferSize>65000</ReceiveBufferSize>
				<ReadBufferSize>65000</ReadBufferSize>
				<SendBufferSize>65000</SendBufferSize>
				<KeepAlive>true</KeepAlive>
				<!-- <TrafficClass>0</TrafficClass> -->
				<!-- <OobInline>false</OobInline> -->
				<!-- <SoLingerTime>-1</SoLingerTime> -->
				<!-- <TcpNoDelay>false</TcpNoDelay> -->
				<AcceptorBackLog>100</AcceptorBackLog>
			</SocketConfiguration>
		</NetConnections>
		<HTTPTunnel>
			<KeepAliveTimeout>2000</KeepAliveTimeout>
		</HTTPTunnel>
		<Client>
			<ClientTimeout>90000</ClientTimeout>
			<IdleFrequency>250</IdleFrequency>
		</Client>
		<!-- RTP/Authentication/Methods defined in Authentication.xml. Default setup includes; none, basic, digest -->
		<RTP>
			<IdleFrequency>75</IdleFrequency>
			<DatagramConfiguration>
				<Incoming>
					<ReuseAddress>true</ReuseAddress>
					<ReceiveBufferSize>1024000</ReceiveBufferSize>
					<SendBufferSize>65000</SendBufferSize>
					<!-- <MulticastBindToAddress>true</MulticastBindToAddress> -->
					<!-- <MulticastInterfaceAddress>192.168.1.22</MulticastInterfaceAddress> -->
					<!-- <TrafficClass>0</TrafficClass> -->
					<MulticastTimeout>50</MulticastTimeout>
					<DatagramMaximumPacketSize>4096</DatagramMaximumPacketSize>
				</Incoming>
				<Outgoing>
					<ReuseAddress>true</ReuseAddress>
					<ReceiveBufferSize>65000</ReceiveBufferSize>
					<SendBufferSize>65000</SendBufferSize>
					<!-- <MulticastBindToAddress>true</MulticastBindToAddress> -->
					<!-- <MulticastInterfaceAddress>192.168.1.22</MulticastInterfaceAddress> -->
					<!-- <TrafficClass>0</TrafficClass> -->
					<MulticastTimeout>50</MulticastTimeout>
					<DatagramMaximumPacketSize>4096</DatagramMaximumPacketSize>
				</Outgoing>
			</DatagramConfiguration>
			<UnicastIncoming>
				<ProcessorCount>8</ProcessorCount>
			</UnicastIncoming>
			<UnicastOutgoing>
				<ProcessorCount>16</ProcessorCount>
			</UnicastOutgoing>
			<MulticastIncoming>
				<ProcessorCount>4</ProcessorCount>
			</MulticastIncoming>
			<MulticastOutgoing>
				<ProcessorCount>4</ProcessorCount>
			</MulticastOutgoing>
		</RTP>
		<Application>
			<ApplicationTimeout>60000</ApplicationTimeout>
			<PingTimeout>12000</PingTimeout>
			<ValidationFrequency>20000</ValidationFrequency>
			<MaximumPendingWriteBytes>0</MaximumPendingWriteBytes>
			<MaximumSetBufferTime>60000</MaximumSetBufferTime>
		</Application>
		<StartStartupStreams>true</StartStartupStreams>
		<!-- Properties defined here will be added to the IVHost.getProperties() collection -->
		<Properties>
		</Properties>
	</VHost>
</Root>

setenv.bat content

@echo off
set _EXECJAVA=java
set JAVA_OPTS=-server -Xmx8000M
set EXECCSCRIPT=cscript
if not exist "%SystemRoot%\SysWOW64\cscript.exe" goto skip64bitcscript
set EXECCSCRIPT="%SystemRoot%\SysWOW64\cscript.exe"
:skip64bitcscript
set WMSCONFIG_URL=
rem set WMSAPP_HOME=C:/wms2
rem set WMSCONFIG_HOME=C:/wms2
set WMSENVOK="false"
set CURRENT_DIR=%cd%
if not "%WMSAPP_HOME%" == "" goto gotAppHome
set WMSAPP_HOME=%CURRENT_DIR%
if exist "%WMSAPP_HOME%\bin\startup.bat" goto okAppHome
cd ..
set WMSAPP_HOME=%cd%
cd %CURRENT_DIR%
:gotAppHome
if exist "%WMSAPP_HOME%\bin\startup.bat" goto okAppHome
%EXECSCRIPT% "displaymsg.vbs" "The WMSAPP_HOME environment variable is missing or not defined correctly (%WMSAPP_HOME%). This environment variable is needed to run the server."
goto end
:okAppHome
rem Guess WMSCONFIG_HOME if not defined
set CURRENT_DIR=%cd%
if not "%WMSCONFIG_HOME%" == "" goto gotConfigHome
set WMSCONFIG_HOME=%WMSAPP_HOME%
if exist "%WMSCONFIG_HOME%\conf\Server.license" goto okConfigHome
cd ..
set WMSCONFIG_HOME=%cd%
cd %CURRENT_DIR%
:gotConfigHome
if exist "%WMSCONFIG_HOME%\conf\Server.license" goto okConfigHome
%EXECSCRIPT% "displaymsg.vbs" "The WMSCONFIG_HOME environment variable is missing or not defined correctly (%WMSCONFIG_HOME%). This environment variable is needed to run the server."
goto end
:okConfigHome
%EXECSCRIPT% "checkjavahome.vbs" //Nologo
if not %errorlevel% == 0 goto end
set WMSENVOK="true"
set _EXECJAVA=%JAVA_HOME%\bin\java.exe
:end

and the transcoder transrate settings (identical for both ingested streams, which come from a Wirecast encoder at the source bitrate, fixed keyframe spacing, etc). We ingest 720p @ about 1.9mbps, pass that through as source stream. The trancoder creates 3 lower bitrates at about 950kbps, 448kbps and 290kbps. CPU load when no clients are connected is about 25-30% with two streams being tanscoded.

<!-- Example template for transrate, producing four new streams at different bitrates. Resultant streams can be played back individually or as a group. source, 360p and 160p encode blocks are enabled through the Enable property, other examples are not enabled. Add additional encode blocks to your template as needed. -->
<Root>
	<Transcode>
		<Encodes>
			<!-- Example Encode block for source, not required unless Member of StreamNameGroup. --> 
			<Encode>
				<Enable>true</Enable>
				<Name>source</Name>
				<StreamName>mp4:${SourceStreamName}_source</StreamName>
				<Video>
					<!-- H.264, PassThru, Disable -->
					<Codec>PassThru</Codec>
					<Bitrate>${SourceVideoBitrate}</Bitrate>
					<Parameters>
					</Parameters>
				</Video>
				<Audio>
					<!-- AAC, PassThru, Disable -->
					<Codec>PassThru</Codec>
					<Bitrate>${SourceAudioBitrate}</Bitrate>
				</Audio>
				<Properties>
				</Properties>
			</Encode>
			<!-- Setup for 720p, high bandwith, main profile for desktop or set-top box -->
			<Encode>
				<Enable>false</Enable>
				<Name>720p</Name>
				<StreamName>mp4:${SourceStreamName}_720p</StreamName>
				<Video>
					<!-- H.264, PassThru, Disable -->
					<Codec>H.264</Codec>
					<!-- default, CUDA, QuickSync -->
					<Transcoder>default</Transcoder>
					<GPUID>-1</GPUID>
					<FrameSize>
						<!-- letterbox, fit-width, fit-height, crop, stretch, match-source -->
						<FitMode>fit-height</FitMode>
						<Width>1280</Width>
						<Height>720</Height>
						<!-- <Crop>0,0,0,0</Crop> -->
						<!-- <SourceRectangle>0,0,320,240</SourceRectangle> -->
					</FrameSize>
					<Profile>main</Profile>
					<Bitrate>1300000</Bitrate>
					<KeyFrameInterval>
						<FollowSource>true</FollowSource>
						<Interval>60</Interval>
					</KeyFrameInterval>
					<Overlays>
						<Overlay>
							<Enable>false</Enable>
							<Index>0</Index>
							<ImagePath>${com.wowza.wms.context.VHostConfigHome}/content/wowzalogo.png</ImagePath>
							<Opacity>100</Opacity>
							<Location>
								<X>5</X>
								<Y>5</Y>
								<Width>${ImageWidth}</Width>
								<Height>${ImageHeight}</Height>
								<!-- horiz: left, right, hcenter - vert: top, bottom, vcenter -->
								<Align>left,top</Align>
							</Location>
						</Overlay>
					</Overlays>
					<Parameters>
					</Parameters>
				</Video>
				<Audio>
					<!-- AAC, PassThru, Disable -->
					<Codec>PassThru</Codec>
					<Bitrate>${SourceAudioBitrate}</Bitrate>
				</Audio>
				<Properties>
				</Properties>
			</Encode>
			<!-- Setup for 360p, high bandwith, main profile for desktop -->
			<Encode>
				<Enable>true</Enable>
				<Name>360p</Name>
				<StreamName>mp4:${SourceStreamName}_360p</StreamName>
				<Video>
					<!-- H.264, PassThru, Disable -->
					<Codec>H.264</Codec>
					<!-- default, CUDA, QuickSync -->
					<Transcoder>default</Transcoder>
					<GPUID>-1</GPUID>
					<FrameSize>
						<!-- letterbox, fit-width, fit-height, crop, stretch, match-source -->
						<FitMode>fit-height</FitMode>
						<Width>640</Width>
						<Height>360</Height>
						<!-- <Crop>0,0,0,0</Crop> -->
						<!-- <SourceRectangle>0,0,320,240</SourceRectangle> -->
					</FrameSize>
					<Profile>main</Profile>
					<Bitrate>850000</Bitrate>
					<KeyFrameInterval>
						<FollowSource>true</FollowSource>
						<Interval>60</Interval>
					</KeyFrameInterval>
					<Overlays>
						<Overlay>
							<Enable>false</Enable>
							<Index>0</Index>
							<ImagePath>${com.wowza.wms.context.VHostConfigHome}/content/wowzalogo.png</ImagePath>
							<Opacity>100</Opacity>
							<Location>
								<X>5</X>
								<Y>5</Y>
								<Width>${ImageWidth}</Width>
								<Height>${ImageHeight}</Height>
								<!-- horiz: left, right, hcenter - vert: top, bottom, vcenter -->
								<Align>left,top</Align>
							</Location>
						</Overlay>
					</Overlays>
					<Parameters>
					</Parameters>
				</Video>
				<Audio>
					<!-- AAC, PassThru, Disable -->
					<Codec>PassThru</Codec>
					<Bitrate>${SourceAudioBitrate}</Bitrate>
				</Audio>
				<Properties>
				</Properties>
			</Encode>
			<!-- Setup for 240p, medium bandwidth, baseline profile -->
			<Encode>
				<Enable>true</Enable>
				<Name>240p</Name>
				<StreamName>mp4:${SourceStreamName}_240p</StreamName>
				<Video>
					<!-- H.264, PassThru, Disable -->
					<Codec>H.264</Codec>
					<!-- default, CUDA, QuickSync -->
					<Transcoder>default</Transcoder>
					<GPUID>-1</GPUID>
					<FrameSize>
						<!-- letterbox, fit-width, fit-height, crop, stretch, match-source  -->
						<FitMode>fit-height</FitMode>
						<Width>360</Width>
						<Height>240</Height>
						<!-- <Crop>0,0,0,0</Crop> -->
						<!-- <SourceRectangle>0,0,320,240</SourceRectangle> -->
					</FrameSize>
					<Profile>baseline</Profile>
					<Bitrate>350000</Bitrate>
					<KeyFrameInterval>
						<FollowSource>true</FollowSource>
						<Interval>60</Interval>
					</KeyFrameInterval>
					<Overlays>
						<Overlay>
							<Enable>false</Enable>
							<Index>0</Index>
							<ImagePath>${com.wowza.wms.context.VHostConfigHome}/content/wowzalogo.png</ImagePath>
							<Opacity>100</Opacity>
							<Location>
								<X>5</X>
								<Y>5</Y>
								<Width>${ImageWidth}</Width>
								<Height>${ImageHeight}</Height>
								<!-- horiz: left, right, hcenter - vert: top, bottom, vcenter -->
								<Align>left,top</Align>
							</Location>
						</Overlay>
					</Overlays>
					<Parameters>
					</Parameters>
				</Video>
				<Audio>
					<!-- AAC, PassThru, Disable -->
					<Codec>PassThru</Codec>
					<Bitrate>${SourceAudioBitrate}</Bitrate>
				</Audio>
				<Properties>
				</Properties>
			</Encode>
			<!-- Setup for 160p, low bandwith, baseline profile for 3G mobile devices such as iOS, Android, Blackberry -->
			<Encode>
				<Enable>true</Enable>
				<Name>160p</Name>
				<StreamName>mp4:${SourceStreamName}_160p</StreamName>
				<Video>
					<!-- H.264, PassThru, Disable -->
					<Codec>H.264</Codec>
					<!-- default, CUDA, QuickSync -->
					<Transcoder>default</Transcoder>
					<GPUID>-1</GPUID>
					<FrameSize>
						<!-- letterbox, fit-width, fit-height, crop, stretch, match-source  -->
						<FitMode>fit-height</FitMode>
						<Width>284</Width>
						<Height>160</Height>
						<!-- <Crop>0,0,0,0</Crop> -->
						<!-- <SourceRectangle>0,0,320,240</SourceRectangle> -->
					</FrameSize>
					<Profile>baseline</Profile>
					<Bitrate>200000</Bitrate>
					<KeyFrameInterval>
						<FollowSource>true</FollowSource>
						<Interval>60</Interval>
					</KeyFrameInterval>
					<Overlays>
						<Overlay>
							<Enable>false</Enable>
							<Index>0</Index>
							<ImagePath>${com.wowza.wms.context.VHostConfigHome}/content/wowzalogo.png</ImagePath>
							<Opacity>100</Opacity>
							<Location>
								<X>5</X>
								<Y>5</Y>
								<Width>${ImageWidth}</Width>
								<Height>${ImageHeight}</Height>
								<!-- horiz: left, right, hcenter - vert: top, bottom, vcenter -->
								<Align>left,top</Align>
							</Location>
						</Overlay>
					</Overlays>
					<Parameters>
					</Parameters>
				</Video>
				<Audio>
					<!-- AAC, PassThru, Disable -->
					<Codec>PassThru</Codec>
					<Bitrate>${SourceAudioBitrate}</Bitrate>
				</Audio>
				<Properties>
				</Properties>
			</Encode>
		</Encodes>
		<Decode>
			<Video>
				<Deinterlace>false</Deinterlace>
				<Overlays>
					<Overlay>
						<Enable>false</Enable>
						<Index>0</Index>
						<ImagePath>${com.wowza.wms.context.VHostConfigHome}/content/wowzalogo.png</ImagePath>
						<Opacity>100</Opacity>
						<Location>
							<X>5</X>
							<Y>5</Y>
							<Width>${ImageWidth}</Width>
							<Height>${ImageHeight}</Height>
							<!-- horiz: left, right, hcenter - vert: top, bottom, vcenter -->
							<Align>left,top</Align>
						</Location>
					</Overlay>
				</Overlays>
				<Parameters>
				</Parameters>
			</Video>
			<Properties>
			</Properties>
		</Decode>
		<StreamNameGroups>
			<!-- Note: Play stream using stream name ngrp:[stream-name] -->
			<StreamNameGroup>
				<Name>${SourceStreamName}_all</Name>
				<Members>
					<Member>
						<EncodeName>source</EncodeName>
					</Member>
					<Member>
						<EncodeName>360p</EncodeName>
					</Member>
					<Member>
						<EncodeName>240p</EncodeName>
					</Member>
					<Member>
						<EncodeName>160p</EncodeName>
					</Member>
				</Members>
			</StreamNameGroup>
			<StreamNameGroup>
				<Name>${SourceStreamName}_mobile</Name>
				<Members>
					<Member>
						<EncodeName>240p</EncodeName>
					</Member>
					<Member>
						<EncodeName>160p</EncodeName>
					</Member>
				</Members>
			</StreamNameGroup>
		</StreamNameGroups>
		<Properties>
		</Properties>
	</Transcode>
</Root>

the codec details for each stream

<smil>
  <head /> 
	<body>
		<switch>
			<video src="mp4:myStream_source" system-bitrate="1941504" width="1280" height="720">
			<param name="audioBitrate" value="98304" valuetype="data" /> 
			<param name="videoBitrate" value="1843200" valuetype="data" /> 
			<param name="videoCodecId" value="avc1.77.31" valuetype="data" /> 
			<param name="audioCodecId" value="mp4a.40.2" valuetype="data" /> 
			</video>
			<video src="mp4:myStream_360p" system-bitrate="948304" width="640" height="360">
			<param name="audioBitrate" value="98304" valuetype="data" /> 
			<param name="videoBitrate" value="850000" valuetype="data" /> 
			<param name="videoCodecId" value="avc1.77.31" valuetype="data" /> 
			<param name="audioCodecId" value="mp4a.40.2" valuetype="data" /> 
			</video>
			<video src="mp4:myStream_240p" system-bitrate="448304" width="426" height="240">
			<param name="audioBitrate" value="98304" valuetype="data" /> 
			<param name="videoBitrate" value="350000" valuetype="data" /> 
			<param name="videoCodecId" value="avc1.66.21" valuetype="data" /> 
			<param name="audioCodecId" value="mp4a.40.2" valuetype="data" /> 
			</video>
			<video src="mp4:myStream_160p" system-bitrate="298304" width="284" height="160">
			<param name="audioBitrate" value="98304" valuetype="data" /> 
			<param name="videoBitrate" value="200000" valuetype="data" /> 
			<param name="videoCodecId" value="avc1.66.21" valuetype="data" /> 
			<param name="audioCodecId" value="mp4a.40.2" valuetype="data" /> 
			</video>
		</switch>
	</body>
</smil>

This is what my player looked like once we hit 1gbps throughput - before I usually see 70-100mbps available and stay locked in the quality that corresponds to screen size. It shows how much of a roller coaster the bandwidth was, and it was doing this consistently, probably as people dropped, players would grab what’s available and then starve again

CPU load went up to 45% somteimes, but I noticed that once we hit the 1gbps brick wall the first core went to 100% quite frequently and I am not sure how to explain that. All CPU used was by java.exe in the processes tab

the java console showed me these graphs

and memory in detail looked like this - the tall central bar in the small heap graph on the bottom hit 100% quite frequently, but then dropped again

I know this is a lot of stuff - any thoughts on possible config errors? IT all ran fine, and it felt like we had no server issues other than the bandwidth being lower for clients than it could have been. that 100% first CPU core has me worried somewhat, though. We moved 1.6TB in a few hours, so things were working for a lot of users:

Server-to-client stream bytes

1607.83 G

Streams

18041

Events

124,699

Unique client IPs

5,683

Server-to-client bytes

1617.23 G

Client-to-server bytes

54.42 G

Hi,

The VHost.xml file is not tuned to take into account the CPUs and threads available.

See the tuning guide here.

https://www.wowza.com/docs/general-tuning

Specifically what jumps out is

120

80

should be

480

360

There may be other sections, but overall tuning is key. You may also need to tweak TCP parameters within Windows OS to get the best delivery as well, but this is something to investigate.

Andrew.

If a problem is ongoing, please describe the details of the problem you are having.

Richard

I’ll make those changes - overlooked that when reading the page, I guess. But you do agree that this isnt why we hit some magical brick wall at 1gbps? Connection and disconnection was fine, server response was fine, just no bandwidth past that mark, which I don’t think has something to do with the thread count. would it have been the reason for the first CPU being overloaded?

based on monitoring tools, the traffic was on the proper interface. Windows also showed it there in the resource monitor and we have ORCA data that usually shows Wowza traffic on that interface.

we have one of these devices available I was just told - http://www.breakingpointsystems.com/products/product-line/breakingpoint-storm/

does this lend iself to load testing Wowza? I can’t find documentation that lists everything it supports.

From the description of the device it appears it does not support any streaming protocols, RTMP, RTSP, HLS, so it wont be able to do what you need.

You can get the Wowza load test tool, which does RTMP, from here

Load test a Wowza Streaming Engine server

You will need multiple servers to act as clients pointing to your main server.

Andrew.

Andrew,

from the documentation of the Storm unit - it is sketchy at best, but there seems to be some support for streaming:

The Storm includes the following example flows:

ClientSim HLS: HTTP Live Streaming (HLS) Client Simulator

RTMP: This superflow simulates RTMP

ClientSim RTSP: This Super Flow starts and ends an RTSP session to a remote server. The media resource name is “example.mpg” and consists of two media streams, one audio and one video. The control names of these streams are assumed to be enumerated in the session description obtained in the server’s DESCRIBE response.

If this means we can generate a set of live clients to put a real load on the server isn’t clear yet. I’ll take a look into this first.

hi Andrew,

You returned as a value:

360 </ PoolSize>

but in the tuning guide will is written:

TransportThreadPool / PoolSize: (40x [total-core-count]) (maximum of 320)

which value is correct? 360 or 320?

you can exceed 320 ?

thanks

steve

Hi,

It actually suggests perhaps the bandwidth was not being pushed out of the 10Gbps interface but perhaps one or more of the 1gbps interfaces. Obviously impossible to tell but if it were just a brick wall traffic scenario then it does suggest more network than thread based.

Usually when threads are lacking you will see errors relating to timecodes, timeouts etc rather than obvious bandwidth limitations.

Worth checking the inbound routing and outbound routing were pointing to the same interface, as if traffic is async then you might find issues. So although your routing pointing inbound to your 10gbps interface, traffic was actually leaving one of the 1gbps interfaces on the server, ie. default route was wrong.

route print

should show where the defaults are etc. It is a common issue when multihoming which can cause issues.

Andrew.

From the description of the device it appears it does not support any streaming protocols, RTMP, RTSP, HLS, so it wont be able to do what you need.

You can get the Wowza load test tool, which does RTMP, from here

https://www.wowza.com/docs/how-to-get-flash-rtmp-load-test-tool

You will need multiple servers to act as clients pointing to your main server.

Andrew.

Hi,

A typo from me there and it should be 320.

Andrew.

Hi,

A typo from me there and it should be 320.

Andrew.

Was this issue resolved? What was the fix? I’m having a similar issue.