Load Testing Tool dropping Connections

Wowza Streaming Engine
1

Hi,

I’m trying so stress-test our server, but I have the phenomenon that clients keep dropping.

I have a standalone Wowza 4 server, connected with 2GBit/s. (Windows Server 2008 R2)

I set up a livestream (800kbit) and start load testing from Amazon EC2 instances (m3.xlarge) using the Wowza Load Testing Tool. I set up workerCount 100 and the Wowza server shows 100 concurrent connections, but after 1-2 minutes the counter keeps dropping and goes down to 80/90.

The client still shows:

INFO server comment - Status running:100 currBehind:0

but the access log shows that it disconnected some clients: (client connectionClosed / pingtimeout)

2014-07-22      17:25:03        CEST    comment session INFO    401     1802720548      client connectionClosed [1802720548] pingtimeout        _defaultVHost_  live    _definst_       603.259 [any]   1935    rtmp://<wowzaserver>:1935/live     54.195.96.8     rtmp    -       WIN 9,0,124,2   1802720548      4230    59508220        -       -       -       -       -       -       -       -       -       -       -       -       -       rtmp://<wowzaserver>:1935/live     -
2014-07-22      17:25:03        CEST    stop    stream  INFO    200     stream_live_de     -       _defaultVHost_  live    _definst_       602.726 [any]   1935    rtmp://<wowzaserver>:1935/live     54.195.96.8     rtmp    -       WIN 9,0,124,2   1802720548      4230    59508220        1       582954  0       59176298        stream_live_de     -       -       -       -       -       rtmp://<wowzaserver>:1935/live/stream_live_de rtmp://<wowzaserver>:1935/live/stream_live_de -       rtmp://<wowzaserver>:1935/live     -
2014-07-22      17:25:03        CEST    destroy stream  INFO    200     stream_live_de     -       _defaultVHost_  live    _definst_       602.726 [any]   1935    rtmp://<wowzaserver>:1935/live     54.195.96.8     rtmp    -       WIN 9,0,124,2   1802720548      4230    59508220        1       -       0       59176298        stream_live_de     -       -       -       -       -       rtmp://<wowzaserver>:1935/live/stream_live_de rtmp://<wowzaserver>:1935/live/stream_live_de -       rtmp://<wowzaserver>:1935/live     -
2014-07-22      17:25:03        CEST    disconnect      session INFO    200     1802720548      -       _defaultVHost_  live    _definst_       603.261 [any]   1935    rtmp://<wowzaserver>:1935/live     54.195.96.8     rtmp    -       WIN 9,0,124,2   1802720548      4230    59508220        -       -       -       -       -       -       -       -       -       -       -       -       -       rtmp://<wowzaserver>:1935/live     -
2014-07-22      17:25:04        CEST    comment session INFO    401     1770218011      client connectionClosed [1770218011] pingtimeout        _defaultVHost_  live    _definst_       524.948 [any]   1935    rtmp://<wowzaserver>:1935/live     54.195.216.243  rtmp    -       WIN 9,0,124,2   1770218011      4162    53965149        -       -       -       -       -       -       -       -       -       -       -       -       -       rtmp://<wowzaserver>:1935/live     -
2014-07-22      17:25:04        CEST    stop    stream  INFO    200     stream_live_de     -       _defaultVHost_  live    _definst_       524.417 [any]   1935    rtmp://<wowzaserver>:1935/live     54.195.216.243  rtmp    -       WIN 9,0,124,2   1770218011      4162    53965149        1       503360  0       53665326        stream_live_de     -       -       -       -       -       rtmp://<wowzaserver>:1935/live/stream_live_de rtmp://<wowzaserver>:1935/live/stream_live_de -       rtmp://<wowzaserver>:1935/live     -
2014-07-22      17:25:04        CEST    destroy stream  INFO    200     stream_live_de     -       _defaultVHost_  live    _definst_       524.418 [any]   1935    rtmp://<wowzaserver>:1935/live     54.195.216.243  rtmp    -       WIN 9,0,124,2   1770218011      4162    53965149        1       -       0       53665326        stream_live_de     -       -       -       -       -       rtmp://<wowzaserver>:1935/live/stream_live_de rtmp://<wowzaserver>:1935/live/stream_live_de -       rtmp://<wowzaserver>:1935/live     -
2014-07-22      17:25:04        CEST    disconnect      session INFO    200     1770218011      -       _defaultVHost_  live    _definst_       524.951 [any]   1935    rtmp://<wowzaserver>:1935/live     54.195.216.243  rtmp    -       WIN 9,0,124,2   1770218011      4162    53965149        -       -       -       -       -       -       -       -       -       -       -       -       -       rtmp://<wowzaserver>:1935/live     -
2014-07-22      17:25:19        CEST    comment session INFO    401     1756336423      client connectionClosed [1756336423] pingtimeout        _defaultVHost_  live    _definst_       619.072 [any]   1935    rtmp://<wowzaserver>:1935/live     54.195.96.8     rtmp    -       WIN 9,0,124,2   1756336423      4298    64203354        -       -       -       -       -       -       -       -       -       -       -       -       -       rtmp://<wowzaserver>:1935/live     -
2014-07-22      17:25:19        CEST    stop    stream  INFO    200     stream_live_de     -       _defaultVHost_  live    _definst_       618.891 [any]   1935    rtmp://<wowzaserver>:1935/live     54.195.96.8     rtmp    -       WIN 9,0,124,2   1756336423      4298    64203354        1       600427  0       63846372        stream_live_de     -       -       -       -       -       rtmp://<wowzaserver>:1935/live/stream_live_de rtmp://<wowzaserver>:1935/live/stream_live_de -       rtmp://<wowzaserver>:1935/live     -
2014-07-22      17:25:19        CEST    destroy stream  INFO    200     stream_live_de     -       _defaultVHost_  live    _definst_       618.893 [any]   1935    rtmp://<wowzaserver>:1935/live     54.195.96.8     rtmp    -       WIN 9,0,124,2   1756336423      4298    64203354        1       -       0       63846372        stream_live_de     -       -       -       -       -       rtmp://<wowzaserver>:1935/live/stream_live_de rtmp://<wowzaserver>:1935/live/stream_live_de -       rtmp://<wowzaserver>:1935/live     -
2014-07-22      17:25:19        CEST    disconnect      session INFO    200     1756336423      -       _defaultVHost_  live    _definst_       619.076 [any]   1935    rtmp://<wowzaserver>:1935/live     54.195.96.8     rtmp    -       WIN 9,0,124,2   1756336423      4298    64203354        -       -       -       -       -       -       -       -       -       -       -       -       -       rtmp://<wowzaserver>:1935/live     -
2014-07-22      17:25:19        CEST    comment session INFO    401     1619303711      client connectionClosed [1619303711] pingtimeout        _defaultVHost_  live    _definst_       619.25  [any]   1935    rtmp://<wowzaserver>:1935/live     54.195.96.8     rtmp    -       WIN 9,0,124,2   1619303711      4298    64813788        -       -       -       -       -       -       -       -       -       -       -       -       -       rtmp://<wowzaserver>:1935/live     -
2014-07-22      17:25:19        CEST    comment session INFO    401     261625871       client connectionClosed [261625871] pingtimeout _defaultVHost_  live    _definst_       619.199 [any]   1935    rtmp://<wowzaserver>:1935/live     54.195.96.8     rtmp    -       WIN 9,0,124,2   261625871       4298    64813788        -       -       -       -       -       -       -       -       -       -       -       -       -       rtmp://<wowzaserver>:1935/live     -
2014-07-22      17:25:19        CEST    stop    stream  INFO    200     stream_live_de     -       _defaultVHost_  live    _definst_       619.019 [any]   1935    rtmp://<wowzaserver>:1935/live     54.195.96.8     rtmp    -       WIN 9,0,124,2   1619303711      4298    64813788        1       605632  0       64453733        stream_live_de     -       -       -       -       -       rtmp://<wowzaserver>:1935/live/stream_live_de rtmp://<wowzaserver>:1935/live/stream_live_de -       rtmp://<wowzaserver>:1935/live     -
2014-07-22      17:25:19        CEST    stop    stream  INFO    200     stream_live_de     -       _defaultVHost_  live    _definst_       619.006 [any]   1935    rtmp://<wowzaserver>:1935/live     54.195.96.8     rtmp    -       WIN 9,0,124,2   261625871       4298    64813788        1       605632  0       64453733        stream_live_de     -       -       -       -       -       rtmp://<wowzaserver>:1935/live/stream_live_de rtmp://<wowzaserver>:1935/live/stream_live_de -       rtmp://<wowzaserver>:1935/live     -
2014-07-22      17:25:19        CEST    destroy stream  INFO    200     stream_live_de     -       _defaultVHost_  live    _definst_       619.02  [any]   1935    rtmp://<wowzaserver>:1935/live     54.195.96.8     rtmp    -       WIN 9,0,124,2   1619303711      4298    64813788        1       -       0       64453733        stream_live_de     -       -       -       -       -       rtmp://<wowzaserver>:1935/live/stream_live_de rtmp://<wowzaserver>:1935/live/stream_live_de -       rtmp://<wowzaserver>:1935/live     -

Is there any way to prevent this? After a few minutes the workerCount decreased for about 50%, which makes it difficult to thoroughly stress-test the server.

Bandwidth can’t be the problem, because I can start several EC2 instances and it doesn’t matter if I start with a workerCount of 50 or 250, the connections will start dropping. The server is <10% of network usage and the EC2 instances should be able to handle ~100Mbit/s.

Any ideas?

I found a similar thread here, but it’s more than 2 years old and wasn’t solved…

2

Hi,

Can you describe where you are running the load test tool? Is this local to you and what is the network bandwidth that you have between the load test instance and the EC2 instance? Simulating 100 clients with an 800kbit/sec stream could saturate a 100mb/sec link.

Paul

3

Hi,

Thanks for the additional information but I think we will need to see both the configuration and log files to troubleshoot this issue. Please create a ticket by emailing support@wowza.com

Include a brief description of the issue, a link to this thread for reference and a zip file containing the following directories:

[Wowza-Install]/conf/

[Wowza-Install]/logs/

[Wowza-Install]/transcoder/

[Wowza-Install]/manager/logs/

Regards,

Jason

4

Hi,

This is now being handled in ticket #98529.

Regards,

Jason

5

Hi,

the load test tool runs on m3.xlarge EC2 instances, which should be capable of handling more than 100Mbit/s. I ran a short test with iperf and between the server and the EC2-instance I reached a throughput of 600 Mbit/s. (with 50 parallel connections)

However, I did also some testing with flazr and with this tool I have the same phenomenon:

08:17:18,976 [New I/O client worker #1-1] INFO [FlvWriter] - write progress: 80 seconds
08:17:18,981 [New I/O client worker #1-4] INFO [FlvWriter] - write progress: 80 seconds
08:17:19,025 [New I/O client worker #1-3] INFO [FlvWriter] - write progress: 80 seconds
08:17:19,027 [New I/O client worker #1-2] INFO [FlvWriter] - write progress: 80 seconds
08:17:19,031 [New I/O client worker #1-3] INFO [FlvWriter] - write progress: 80 seconds
08:17:19,032 [New I/O client worker #1-4] INFO [FlvWriter] - write progress: 80 seconds
08:17:20,442 [New I/O client worker #1-1] INFO [ClientHandler] - channel closed: [id: 0x2b94d9d5, /10.96.26.61:58637 => <WowzaServer>/<WowzaServerIp>:1935] CLOSED
08:17:20,442 [New I/O client worker #1-1] INFO [FlvWriter] - finished in 82 seconds, media duration: 51 seconds (seek time: 0)
08:17:20,507 [New I/O client worker #1-2] INFO [ClientHandler] - channel closed: [id: 0x1795f1cc, /10.96.26.61:58566 => <WowzaServer>/<WowzaServerIp>:1935] CLOSED
08:17:20,507 [New I/O client worker #1-2] INFO [FlvWriter] - finished in 82 seconds, media duration: 52 seconds (seek time: 0)

Wowza Accesslog:

2014-07-24      10:16:58        CEST    comment session INFO    401     1123310546      client connectionClosed [1123310546] pingtimeout        _defaultVHost_  live        _definst_       75.531  [any]   1935    rtmp://<WowzaServer>:1935/live 54.75.117.96    rtmp    -       WIN 9,0,124,2   1123310546      3524    6663243 -       -       -       -       -       -       -       -       -       -       -       -       -       rtmp://<WowzaServer>:1935/live -
2014-07-24      10:16:58        CEST    stop    stream  INFO    200     stream_live_de     -       _defaultVHost_  live        _definst_       75.281  [any]   1935    rtmp://<WowzaServer>:1935/live 54.75.117.96    rtmp    -       WIN 9,0,124,2   1123310546      3524    6663243 1       58368   0       6624594 stream_live_de     -       -       -       -       -       rtmp://<WowzaServer>:1935/live/stream_live_de     rtmp://<WowzaServer>:1935/live/stream_live_de     -       rtmp://<WowzaServer>:1935/live -
2014-07-24      10:16:58        CEST    destroy stream  INFO    200     stream_live_de     -       _defaultVHost_  live        _definst_       75.282  [any]   1935    rtmp://<WowzaServer>:1935/live 54.75.117.96    rtmp    -       WIN 9,0,124,2   1123310546      3524    6663243 1       -       0       6624594 stream_live_de     -       -       -       -       -       rtmp://<WowzaServer>:1935/live/stream_live_de     rtmp://<WowzaServer>:1935/live/stream_live_de     -       rtmp://<WowzaServer>:1935/live -
2014-07-24      10:16:58        CEST    disconnect      session INFO    200     1123310546      -       _defaultVHost_  live        _definst_       75.533  [any]   1935    rtmp://<WowzaServer>:1935/live 54.75.117.96    rtmp    -       WIN 9,0,124,2   1123310546      3524    6663243 -       -       -       -       -       -       -       -       -       -       -       -       -       rtmp://<WowzaServer>:1935/live -
2014-07-24      10:16:58        CEST    comment session INFO    401     996155186       client connectionClosed [996155186] pingtimeout _defaultVHost_  live        _definst_       75.742  [any]   1935    rtmp://<WowzaServer>:1935/live 54.75.117.96    rtmp    -       WIN 9,0,124,2   996155186       3524    6630915 -       -       -       -       -       -       -       -       -       -       -       -       -       rtmp://<WowzaServer>:1935/live -
2014-07-24      10:16:58        CEST    stop    stream  INFO    200     stream_live_de     -       _defaultVHost_  live        _definst_       75.527  [any]   1935    rtmp://<WowzaServer>:1935/live 54.75.117.96    rtmp    -       WIN 9,0,124,2   996155186       3524    6630915 1       57963   0       6592484 stream_live_de     -       -       -       -       -       rtmp://<WowzaServer>:1935/live/stream_live_de     rtmp://<WowzaServer>:1935/live/stream_live_de     -       rtmp://<WowzaServer>:1935/live -
2014-07-24      10:16:58        CEST    destroy stream  INFO    200     stream_live_de     -       _defaultVHost_  live        _definst_       75.527  [any]   1935    rtmp://<WowzaServer>:1935/live 54.75.117.96    rtmp    -       WIN 9,0,124,2   996155186       3524    6630915 1       -       0       6592484 stream_live_de     -       -       -       -       -       rtmp://<WowzaServer>:1935/live/stream_live_de     rtmp://<WowzaServer>:1935/live/stream_live_de     -       rtmp://<WowzaServer>:1935/live -
2014-07-24      10:16:58        CEST    disconnect      session INFO    200     996155186       -       _defaultVHost_  live        _definst_       75.744  [any]   1935    rtmp://<WowzaServer>:1935/live 54.75.117.96    rtmp    -       WIN 9,0,124,2   996155186       3524    6630915 -       -       -       -       -       -       -       -       -       -       -       -       -       rtmp://<WowzaServer>:1935/live -