To whom It may concern;
I have an up and running operation with an average of 500 people streaming all the time, this number is gradually increasing.
I have an origin and an edge server (all centos based) and I am using the load balancer plugin version 4.0 on the edge server as both “Server and Client”
The edge server has 2 x 1Gbps bonded as bond1 so it should support up to 1.8 Gbps in concurrent incoming traffic.
Now the issue that’s been happening recently is that during peak-hours, 1500+ concurrent connections the whole service is disrupted and I come to find the load balancer info interface displaying a large number of connections but a bandwidth of 11000 Kbps or so, and the worst part is that even if we restart wowza, the network, or even the whole server, the issue remains the same. Noting that the peak traffic the user is reaching is around 500 Mbps before this issue starts to appear.
When further investing, and after restarting we found such irregularities in the logs :
2016-11-05 18:29:24 WAT comment server INFO 200 - LB XMLParser: xmlParse() Timed out posting to http://127.0.0.1:1999/loadbalancerInterface?command=AAAAq0MzY5PvqNyULp9nwZ8SvDFs41BeB3z9bX1uKa7Z98zk1EHl1QKCO83UeDhDXHFxkdRB5dUC%250AgjvNqgrFtbPMicLGgWV9JF37Mx1ZZSkkCGdUyeYY2f3e9TSKr4sFT4QGtDcu7Nx4Kjdtk2YWAX%252F5%250AL0Nt%252Fu%252FoaOVfHZoM0JDR1MJSk2YWAX%252F5L0MWc0u9HsoQgSBVPX%252Brs7iIfdgwLjYoqpqTg9npey6s%250ACa0x7Qmiar7%252F
Or sometimes this :
2016-11-03 20:33:05 WAT comment server WARN 200 - LoadBalancerPublicInterface.onHTTPRequest() No Load Balancers found when requesting redirect/57878080215cc40
I keep restarting wowza, the network, and the whole server, and on and on again until it starts to work again, I tried all the possible reasoning and possible scalability measures, and nothing works.
Would really appreciate your help on this one.