Load Balancer 4.0 - Least Loaded Edge

[Jordan_Rumph]

Hi everyone,

I have something similar posted (sort of) in the Wowza nDVR forum here;

Wowza 4.1 nDVR with Load Balancer 4.0

Hello, We are trying to upgrade our 8 Wowza 3 servers to Wowza 4, and I am working out any bugs in our development environment to make sure it goes well. I was able to setup Wowza 4.1 with nDVR and Load Balancer 4.0 without any problems. But I have a few questions about it. When I use nDVR and Load Balancer with the Apple HLS (M3U8) redirect method. If I point my client at this for redirection: http://[load-balancer-ip-address]:1935/redirect/[application-name]/[stream-name]?scheme=m3u8 It …

Since that thread dealt mainly with Load Balancer 4.0 and nDVR 4.1, I thought I would open a new one for this specific issue.

Our setup:

Wowza Version: 4.1.0

DVR Version: 4.1

Load Balance Version: 4.0

Server1

liveorigin (primary)

liveedge (points to primary|backup)

Load Balancer Server (primary)

Load Balancer Client

Server2

liveorigin (backup)

liveedge (points to primary|backup)

Load Balancer Server (backup)

Load Balancer Client

We are having our clients (Amino set-top-boxs) go to this address to get the least loaded server with this address;

http://[load-balancer-ip-address]:1935/redirect/?request=server

What seems to be happening is unless we set a connection limit on Server1 (e.g. 50), then all clients get redirected to Server1/liveedge and none to Server2/liveedge. If we set a limit on Server1, then clients will start getting redirected to Server2 once Server1 hits it’s limit. But if we leave limits off (like we would prefer) then only Server1 gets the load. I’m sure I am doing something wrong, but I can’t seem to to figure it out. Here are some relevent files:

Server1 - conf/Server.xml

<?xml version="1.0" encoding="UTF-8"?>
<Root version="2">
<Server>
<Name>Wowza Streaming Engine</Name>
<Description>Wowza Streaming Engine is robust, customizable, and scalable server software that powers reliable streaming of high-quality video and audio to any device, anywhere.</Description>
<RESTInterface>
<Enable>true</Enable>
<IPAddress>*</IPAddress>
<Port>8087</Port>
<!-- none, basic, digest-->
<AuthenticationMethod>digest</AuthenticationMethod>
<DiagnosticURLEnable>true</DiagnosticURLEnable>
<SSLConfig>
<Enable>false</Enable>
<KeyStorePath></KeyStorePath>
<KeyStorePassword></KeyStorePassword>
<KeyStoreType>JKS</KeyStoreType>
<SSLProtocol>TLS</SSLProtocol>
<Algorithm>SunX509</Algorithm>
<CipherSuites></CipherSuites>
<Protocols></Protocols>
</SSLConfig>
<IPWhiteList>127.0.0.1</IPWhiteList>
<IPBlackList></IPBlackList>
<EnableXMLFile>false</EnableXMLFile>
<DocumentationServerEnable>false</DocumentationServerEnable>
<DocumentationServerPort>8089</DocumentationServerPort>
<!-- none, basic, digest-->
<DocumentationServerAuthenticationMethod>digest</DocumentationServerAuthenticationMethod>
<Properties>
</Properties>
</RESTInterface>
<CommandInterface>
<HostPort>
<ProcessorCount>${com.wowza.wms.TuningAuto}</ProcessorCount>
<IpAddress>*</IpAddress>
<Port>8083</Port>
</HostPort>
</CommandInterface>
<AdminInterface>
<!-- Objects exposed through JMX interface: Server, VHost, VHostItem, Application, ApplicationInstance, MediaCaster, Module, Client, MediaStream, SharedObject, Acceptor, IdleWorker -->
<ObjectList>Server,VHost,VHostItem,Application,ApplicationInstance,MediaCaster,Module,IdleWorker</ObjectList>
</AdminInterface>
<Stats>
<Enable>true</Enable>
</Stats>
<!-- JMXUrl: service:jmx:rmi://localhost:8084/jndi/rmi://localhost:8085/jmxrmi -->
<JMXRemoteConfiguration>
<Enable>true</Enable>
<IpAddress>localhost</IpAddress> <!-- set to localhost or internal ip address if behind NAT -->
<!-- Res_616 IP address of this server -->
<RMIServerHostName>change_ip</RMIServerHostName> <!-- set to external ip address or domain name if behind NAT -->
<RMIConnectionPort>8084</RMIConnectionPort>
<RMIRegistryPort>8085</RMIRegistryPort>
<Authenticate>true</Authenticate>
<PasswordFile>${com.wowza.wms.ConfigHome}/conf/jmxremote.password</PasswordFile>
<AccessFile>${com.wowza.wms.ConfigHome}/conf/jmxremote.access</AccessFile>
<SSLSecure>false</SSLSecure>
</JMXRemoteConfiguration>
<UserAgents>Shockwave Flash|CFNetwork|MacNetwork/1.0 (Macintosh)</UserAgents>
<Streams>
<DefaultStreamPrefix>mp4</DefaultStreamPrefix>
</Streams>
<ServerListeners>
<ServerListener>
<BaseClass>com.wowza.wms.plugin.loadbalancer.general.LoadBalancerServer</BaseClass>
</ServerListener>
</ServerListeners>
<VHostListeners>
<!--
<VHostListener>
<BaseClass></BaseClass>
</VHostListener>
-->
</VHostListeners> 
<HandlerThreadPool>
<PoolSize>${com.wowza.wms.TuningAuto}</PoolSize>
</HandlerThreadPool>
<TransportThreadPool>
<PoolSize>${com.wowza.wms.TuningAuto}</PoolSize>
</TransportThreadPool>
<RTP>
<DatagramStartingPort>6970</DatagramStartingPort>
<DatagramPortSharing>false</DatagramPortSharing>
</RTP>
<Manager>
<!-- Properties defined are used by the Manager -->
<Properties>
</Properties>
</Manager>
<!-- Properties defined here will be added to the IServer.getProperties() collection -->
<Properties>
<!--
Specifies the server type in the load-balancing
configuration. The Load Balancer has the value
Server, load-balancing Servers have the value
Client, and servers that perform both roles have
the value Server,Client.
Deployed on [loadbalanceType]: Server,Client
-->
<Property>
<Name>loadbalanceType</Name>
<Value>Server,Client</Value>
<Type>String</Type>
</Property>
<!--
Enables encrypted communication between the
Load Balancer and Servers. The value must be
at least 8 characters long and must be the same
on all servers in the configuration.
Deployed on [loadbalanceType]: Server,Client
-->
<Property>
<Name>loadbalanceKey</Name>
<Value>lb_password</Value>
<Type>String</Type>
</Property>
<!--
Name of the Server that connects to the Load
Balancer. A random name is used if one isn't
provided.
Deployed on [loadbalanceType]: Client
-->
<Property>
<Name>loadbalanceClientName</Name>
<Value>Server1_hostname</Value>
<Type>String</Type>
</Property>
<!--
IP address of the Load Balancer. Use the cluster VIP.
Deployed on [loadbalanceType]: Server,Client
-->
<Property>
<Name>loadbalanceServerIP</Name>
<Value>Server1/Server2_Virtual_IP</Value>
<Type>String</Type>
</Property>
<!--
Port that the Load Balancer runs on. By default,
this should be set to 1935 or to any other port
that's configured in your VHost.xml file.
Deployed on [loadbalanceType]: Server,Client
-->
<Property>
<Name>loadbalanceServerPort</Name>
<Value>1935</Value>
<Type>String</Type>
</Property>
<!--
(Optional) A comma-separated list of
applications to include in bandwidth and
connection load-balancing calculations. Leaving
this value blank includes all applications on the
Server automatically. Any applications specified
in loadbalanceApplicationsExclude override
this setting.
Deployed on [loadbalanceType]: Client
-->
<Property>
<Name>loadbalanceApplicationsInclude</Name>
<Value>liveedge,vod</Value>
<Type>String</Type>
</Property>
<!--
Order in which decision-making is processed for Server
load balancing. Some or all of the following
values can be entered as a comma-separated list:
Geographic – Select load balancing Servers
based on the geographic selector.
Bandwidth – Select load balancing Servers
based on the bandwidth selector.
Connection – Select load balancing Servers
based on the connection selector.
Deployed on [loadbalanceType]: 
-->
<Property>
<Name>loadbalanceDecisionOrder</Name>
<Value>Connection</Value>
<Type>String</Type>
</Property>
<!--
Controls bandwidth monitoring. The default
value is Off. To enable bandwidth monitoring,
set this value to On, and then configure the
loadbalanceBandwidthLimit property.
Deployed on [loadbalanceType]: Client
-->
<Property>
<Name>loadbalanceBandwidthEnable</Name>
<Value>Off</Value>
<Type>String</Type>
</Property>
<!--
The bandwidth limit for this Server, in kilobits
per second (kbps). For example, if you want this
Server to provide only 50 megabits per second
(Mbps) of throughput, set the value to 50000.
To allow unlimited bandwidth, set to 0.
Client
If all Servers in the load-balancing deployment
are saturated, redirection will fail and you must
add an additional load-balancing Server to
increase capacity.
Deployed on [loadbalanceType]: Client
-->
<Property>
<Name>loadbalanceBandwidthLimit</Name>
<Value>0</Value>
<Type>String</Type>
</Property>
<!--
Controls connection balancing. The default
value is Off. To enable connection balancing,
set this value to On, and then configure the
loadbalanceConnectionLimit property.
Deployed on [loadbalanceType]: Client
-->
<Property>
<Name>loadbalanceConnectionEnable</Name>
<Value>On</Value>
<Type>String</Type>
</Property>
<!--
The limit for outgoing connection on this
Server. To allow an unlimited number of
connections, set to 0.
If all Servers in the load-balancing deployment
are servicing the maximum number of
connections, redirection will fail and you must
add an additional load-balancing Server to
increase capacity.
Deployed on [loadbalanceType]: Client
-->
<Property>
<Name>loadbalanceConnectionLimit</Name>
<Value>0</Value>
<Type>String</Type>
</Property>
<!--
A comma-separated list of client names. Clients
with these names can connect to the Load
Balancer and not be redirected. For example,
you can specify the client name FMLE to enable
a client that has Flash Media Encoder installed
to connect.
Deployed on [loadbalanceType]: Server
-->
<Property>
<Name>loadbalanceIgnoreClients</Name>
<Value>FMLE</Value>
<Type>String</Type>
</Property>
<!--
(Optional) Enables load-balancing Servers to be
shut down via the Statistics web interface.
Deployed on [loadbalanceType]: Server
-->
<Property>
<Name>loadbalanceAllowRemoteClientShutdown</Name>
<Value>On</Value>
<Type>String</Type>
</Property>
<!--
(Optional) Controls decision-making debug
logging, which helps you to see how the Load
Balancer obtained a result.
Deployed on [loadbalanceType]: Server
-->
<Property>
<Name>loadbalanceDebugServerDecision</Name>
<Value>Off</Value>
<Type>String</Type>
</Property>
</Properties>
</Server>
</Root>

Server2 - conf/Server.xml

<?xml version="1.0" encoding="UTF-8"?>
<Root version="2">
<Server>
<Name>Wowza Streaming Engine</Name>
<Description>Wowza Streaming Engine is robust, customizable, and scalable server software that powers reliable streaming of high-quality video and audio to any device, anywhere.</Description>
<RESTInterface>
<Enable>true</Enable>
<IPAddress>*</IPAddress>
<Port>8087</Port>
<!-- none, basic, digest-->
<AuthenticationMethod>digest</AuthenticationMethod>
<DiagnosticURLEnable>true</DiagnosticURLEnable>
<SSLConfig>
<Enable>false</Enable>
<KeyStorePath></KeyStorePath>
<KeyStorePassword></KeyStorePassword>
<KeyStoreType>JKS</KeyStoreType>
<SSLProtocol>TLS</SSLProtocol>
<Algorithm>SunX509</Algorithm>
<CipherSuites></CipherSuites>
<Protocols></Protocols>
</SSLConfig>
<IPWhiteList>127.0.0.1</IPWhiteList>
<IPBlackList></IPBlackList>
<EnableXMLFile>false</EnableXMLFile>
<DocumentationServerEnable>false</DocumentationServerEnable>
<DocumentationServerPort>8089</DocumentationServerPort>
<!-- none, basic, digest-->
<DocumentationServerAuthenticationMethod>digest</DocumentationServerAuthenticationMethod>
<Properties>
</Properties>
</RESTInterface>
<CommandInterface>
<HostPort>
<ProcessorCount>${com.wowza.wms.TuningAuto}</ProcessorCount>
<IpAddress>*</IpAddress>
<Port>8083</Port>
</HostPort>
</CommandInterface>
<AdminInterface>
<!-- Objects exposed through JMX interface: Server, VHost, VHostItem, Application, ApplicationInstance, MediaCaster, Module, Client, MediaStream, SharedObject, Acceptor, IdleWorker -->
<ObjectList>Server,VHost,VHostItem,Application,ApplicationInstance,MediaCaster,Module,IdleWorker</ObjectList>
</AdminInterface>
<Stats>
<Enable>true</Enable>
</Stats>
<!-- JMXUrl: service:jmx:rmi://localhost:8084/jndi/rmi://localhost:8085/jmxrmi -->
<JMXRemoteConfiguration>
<Enable>true</Enable>
<IpAddress>localhost</IpAddress> <!-- set to localhost or internal ip address if behind NAT -->
<!-- Res_616 IP address of this server -->
<RMIServerHostName>change_ip</RMIServerHostName> <!-- set to external ip address or domain name if behind NAT -->
<RMIConnectionPort>8084</RMIConnectionPort>
<RMIRegistryPort>8085</RMIRegistryPort>
<Authenticate>true</Authenticate>
<PasswordFile>${com.wowza.wms.ConfigHome}/conf/jmxremote.password</PasswordFile>
<AccessFile>${com.wowza.wms.ConfigHome}/conf/jmxremote.access</AccessFile>
<SSLSecure>false</SSLSecure>
</JMXRemoteConfiguration>
<UserAgents>Shockwave Flash|CFNetwork|MacNetwork/1.0 (Macintosh)</UserAgents>
<Streams>
<DefaultStreamPrefix>mp4</DefaultStreamPrefix>
</Streams>
<ServerListeners>
<ServerListener>
<BaseClass>com.wowza.wms.plugin.loadbalancer.general.LoadBalancerServer</BaseClass>
</ServerListener>
</ServerListeners>
<VHostListeners>
<!--
<VHostListener>
<BaseClass></BaseClass>
</VHostListener>
-->
</VHostListeners> 
<HandlerThreadPool>
<PoolSize>${com.wowza.wms.TuningAuto}</PoolSize>
</HandlerThreadPool>
<TransportThreadPool>
<PoolSize>${com.wowza.wms.TuningAuto}</PoolSize>
</TransportThreadPool>
<RTP>
<DatagramStartingPort>6970</DatagramStartingPort>
<DatagramPortSharing>false</DatagramPortSharing>
</RTP>
<Manager>
<!-- Properties defined are used by the Manager -->
<Properties>
</Properties>
</Manager>
<!-- Properties defined here will be added to the IServer.getProperties() collection -->
<Properties>
<!--
Specifies the server type in the load-balancing
configuration. The Load Balancer has the value
Server, load-balancing Servers have the value
Client, and servers that perform both roles have
the value Server,Client.
Deployed on [loadbalanceType]: Server,Client
-->
<Property>
<Name>loadbalanceType</Name>
<Value>Server,Client</Value>
<Type>String</Type>
</Property>
<!--
Enables encrypted communication between the
Load Balancer and Servers. The value must be
at least 8 characters long and must be the same
on all servers in the configuration.
Deployed on [loadbalanceType]: Server,Client
-->
<Property>
<Name>loadbalanceKey</Name>
<Value>lb_password</Value>
<Type>String</Type>
</Property>
<!--
Name of the Server that connects to the Load
Balancer. A random name is used if one isn't
provided.
Deployed on [loadbalanceType]: Client
-->
<Property>
<Name>loadbalanceClientName</Name>
<Value>Server2_hostname</Value>
<Type>String</Type>
</Property>
<!--
IP address of the Load Balancer. Use the cluster VIP.
Deployed on [loadbalanceType]: Server,Client
-->
<Property>
<Name>loadbalanceServerIP</Name>
<Value>Server1/Server2_Virtual_IP</Value>
<Type>String</Type>
</Property>
<!--
Port that the Load Balancer runs on. By default,
this should be set to 1935 or to any other port
that's configured in your VHost.xml file.
Deployed on [loadbalanceType]: Server,Client
-->
<Property>
<Name>loadbalanceServerPort</Name>
<Value>1935</Value>
<Type>String</Type>
</Property>
<!--
(Optional) A comma-separated list of
applications to include in bandwidth and
connection load-balancing calculations. Leaving
this value blank includes all applications on the
Server automatically. Any applications specified
in loadbalanceApplicationsExclude override
this setting.
Deployed on [loadbalanceType]: Client
-->
<Property>
<Name>loadbalanceApplicationsInclude</Name>
<Value>liveedge,vod</Value>
<Type>String</Type>
</Property>
<!--
Order in which decision-making is processed for Server
load balancing. Some or all of the following
values can be entered as a comma-separated list:
Geographic – Select load balancing Servers
based on the geographic selector.
Bandwidth – Select load balancing Servers
based on the bandwidth selector.
Connection – Select load balancing Servers
based on the connection selector.
Deployed on [loadbalanceType]: 
-->
<Property>
<Name>loadbalanceDecisionOrder</Name>
<Value>Connection</Value>
<Type>String</Type>
</Property>
<!--
Controls bandwidth monitoring. The default
value is Off. To enable bandwidth monitoring,
set this value to On, and then configure the
loadbalanceBandwidthLimit property.
Deployed on [loadbalanceType]: Client
-->
<Property>
<Name>loadbalanceBandwidthEnable</Name>
<Value>Off</Value>
<Type>String</Type>
</Property>
<!--
The bandwidth limit for this Server, in kilobits
per second (kbps). For example, if you want this
Server to provide only 50 megabits per second
(Mbps) of throughput, set the value to 50000.
To allow unlimited bandwidth, set to 0.
Client
If all Servers in the load-balancing deployment
are saturated, redirection will fail and you must
add an additional load-balancing Server to
increase capacity.
Deployed on [loadbalanceType]: Client
-->
<Property>
<Name>loadbalanceBandwidthLimit</Name>
<Value>0</Value>
<Type>String</Type>
</Property>
<!--
Controls connection balancing. The default
value is Off. To enable connection balancing,
set this value to On, and then configure the
loadbalanceConnectionLimit property.
Deployed on [loadbalanceType]: Client
-->
<Property>
<Name>loadbalanceConnectionEnable</Name>
<Value>On</Value>
<Type>String</Type>
</Property>
<!--
The limit for outgoing connection on this
Server. To allow an unlimited number of
connections, set to 0.
If all Servers in the load-balancing deployment
are servicing the maximum number of
connections, redirection will fail and you must
add an additional load-balancing Server to
increase capacity.
Deployed on [loadbalanceType]: Client
-->
<Property>
<Name>loadbalanceConnectionLimit</Name>
<Value>0</Value>
<Type>String</Type>
</Property>
<!--
A comma-separated list of client names. Clients
with these names can connect to the Load
Balancer and not be redirected. For example,
you can specify the client name FMLE to enable
a client that has Flash Media Encoder installed
to connect.
Deployed on [loadbalanceType]: Server
-->
<Property>
<Name>loadbalanceIgnoreClients</Name>
<Value>FMLE</Value>
<Type>String</Type>
</Property>
<!--
(Optional) Enables load-balancing Servers to be
shut down via the Statistics web interface.
Deployed on [loadbalanceType]: Server
-->
<Property>
<Name>loadbalanceAllowRemoteClientShutdown</Name>
<Value>On</Value>
<Type>String</Type>
</Property>
<!--
(Optional) Controls decision-making debug
logging, which helps you to see how the Load
Balancer obtained a result.
Deployed on [loadbalanceType]: Server
-->
<Property>
<Name>loadbalanceDebugServerDecision</Name>
<Value>Off</Value>
<Type>String</Type>
</Property>
</Properties>
</Server>
</Root>

We would like for the least loaded server IP to get returned without setting limits on Server1. That way, each server fills up evenly. Thanks for your help!

Jordan

[Jordan_Rumph]

Hi all,

Until I get this resolved, I was able to install the Loadbalancer_2.0 on the Wowza 4.1.1 servers and all works as expected. I get the least loaded edge every time I request it with;

http://load_balancer:1935/loadbalancer

Surely, though, I can get the same functionality with Loadbalancer_4.0?

Thanks for the help!

Jordan

[Artur_Pydo]

I will suggest to make the choice of equally loadbalancing between all the servers or to full load the first server before switching to the next to be a configurable feature.

In this case one could be able to load the preferred servers first or to load them all if all of them are the same.

[Andrew_Kennedy1]

Hi,

It should already work in that manner. The decision based system works as follows

• Server requested

• Choose servers out of the clients available

• If more than one choice is returned - As they provide the same results - ie. same connection counts, same bandwidth, etc - just the first one

The next request should then go to another server

• Updates from clients to the load balancer only occur every 10 seconds. This can be reduced to 5 via configuration

If you turn on the decision debug then this should show how the load balancer came up with the result.

Andrew.

[Andrew_Kennedy1]

Hi,

I will certainly take this feedback internally and look at how we can further test this.

Andrew.

[sal_jefferson]

Hello there,

I found this response from a support team member. It may relate to your findings but I am not 100% certain.

As the new load balancer is relatively new we are collecting feedback both for new features and comparison to the previous load balancer. As we have significantly changed the core of the load balancer some old the older features have been dropped initially but we do have plans to add them back as we further develop it.

You may decide to open a ticket with support@wowza.com to find out for sure.

Kind regards,

Salvadore

[Gaurav_Mishra1]

We are also facing the same issue described above. we configured two servers behind our load balancer and set order choice to Connection. doing so we observed that loadbalancer keeps on returning the first node till it hits Connection limit. if no Connection limit is set of first server then second node is never returned from loadbalancer.

We are also reverting to Loadbalancer 2.0.