My clients keep getting disconnected when trying to broadcast live video. Downstream to the client doesn’t seem to have any problem. I can transmit 10 KB/s but anything beyond 50 KB/s will cause the disconnect problem.
The Wowza access log indicates a ping timeout:
INFO session comment 762278498 client connectionClosed [762278498] pingtimeout
I’ve also seen this in the log:
INFO session connect-pending 98.246.109.194 -
I’m running Wowza 3 on a virtual CentOS 64bit system with 1.75 GB of memory. The Java heap size is set at 500 MB. The application is configured as live low latency with send and receive buffers of 12000. Latency seems ok. Client is flash player 11.
Now I know the above are below the recommended specs for Wowza, but I’d still like to rule out a network problem before upgrading the server, especially since if I have a problem so will my network users. My internet connection is greater than 3 Mb/s down to the client and 4 Mb/s up to the server.
Wowza as configured above + Apache + MySQL + 1 or 2 connected clients + JConsole + ssh will take up between 80-88% of the available memory. CPU usage by the Java process is below 4%. When broadcasting video to the server, other processes (like http to apache) seem sluggish and other connected clients on my network besides the video clients will disconnect (JConsole, even ssh occasionally). Again, streaming down to clients doesn’t seem to be a problem though I haven’t fully tested this yet. Still at peaks up to 300 KB/s down, no problems, but 50 KB/s up will cause the disconnect problem in less than 1 minute of continuous operation.
Any solutions or suggestions for further troubleshooting would be greatly appreciated.
Are you using RTMPT? There are higher number of ping timeout disconnects when using RMTPT iinstead of RTMP
Richard
Right, because it is a network related problem. Wowza keeps tabs on clients and if they disappear Wowza drops its connection
Richard
I have run Wowza on EC2 micro instance and old machines with 256k ram. It’s not recommended, but it will do a minimal amount of streaming.
Many hosting services are designed for web hosting. I’m not sure, what the problem is with rtmp on that network, and we might not, probably won’t, be able to solve that problem, especially if it is by design, e.g., blocked rtmp.
If you are just testing, try an EC2 m1-small. Don’t use the micro, it can work but it’s not adequate
Richard
No, I’m using RTMP. Note that the timeouts do not occur when running the Wowza server locally (Macbook Pro with 4 GB memory) at the default heap size.
The problem with dismissing it as a network problem is repeated speedtests all give me over 4 Mb/s upload speed to the internet. Furthermore, I have no problem doing live video with Skype. If it is a network problem, something between me and my hosting company is blocking me, a task I’m not relishing debugging.
What I want to rule out is that Wowza is running on an underpowered system. Specifically can Wowza run on a virtual Centos 64 bit Linux system with just 1.7 MB? Running on the remote server, I had no trouble with download streaming of over 400 KB/s, but upstream of 50 KB/s disconnects before running just 1 minute. I had to adjust the heap size for the default of 1200 MB to 500 to run on the remote server. Adjusting it to the minimum needed to start (750 MB) causes Wowza to run out of heap space whether it’s downstreaming or upstreaming. The system requirements state that on a linux system you should have at least a quad core with 1 GB RAM per core. With this information do you still think it’s purely a network problem?
I’ve got further evidence that something is blocking incoming transmissions but it’s not my access to the internet. Running iperf3 on the same host as Wowza shows fine connectivity until I start transmitting a live RTMP stream. The rate then goes to down to near zero (and sometimes zero depending on the run). When I run a speedtest to the internet using speedtest.net, the outgoing stream (to the internet) averages around 4 Mb/s. I think these stats prove that I’m not being blocked by my ISP. So possibilities are either Wowza is running on an underpowered server or something in my host provider is blocking me. My host provider claims it’s not them, leaving the former, but it would seem that the first replies would have told me that. Again, Wowza is running on a Centos 64 bit system with 1.75 GB of memory. Wowza is running with a heap size of 500 MB.
Here’s a typical run of iperf3
period action rate (to server)
0 - 10.68s connected, idle 4.81 Mb/s
10.68 - 49.38 s xmit to server 0.84 Mb/s
49.38 - 59.43 s disconnected 4.90 Mb/s
The client transmitting disconnects towards the end of the second period (~40 s). The transmit rate is 125 MB/s (that’s mega btyes, other Mb is Mega bits). Wowza is running with a heap size of 500 MB and the Java virtual machine reports using 1209 MB (out of total of 1750) of memory.
Note that if I downstream, the iperf reported client to server upload stats are statistically the same as are those from speedtest.net. The only difference is the client doesn’t disconnect.
Any ideas?
I believe I found the problem. It was an Ethernet switch that was connected to the cable modem via the AC power line. Normally, it is faster and more reliable than the wireless router, but when I went through the wireless router, the problem disappeared. I’m a little surprised – the ethernet switch is supposed to operate at 100 MB/s and it only had two computers connected to it. One thing that should have tipped me off was that clients running on each of the connected machines tended to disconnect at about the same time even when only one of them was transmitting.