Very high CPU - Wowza 3.0.5

Wowza Streaming Engine
1

I’m load-testing Wowza 3.0.5 on EC2 m1.small instances.

The configuration is:

  • Ubuntu 10.10 32-bit, Java -server 1.7u3

  • standard liverepeater/edge (origin is an m1.large, performance is not a problem on this machine at the moment)

  • live 128kbps mp3 streams

  • no custom modules

  • Wowza tuning guide followed (VHost.xml below)

I’m using rtmpdump on a third server to slowly build up outgoing connections to the edge. CPU usage on the edge server increases steadily (faster than I would expect to be honest) but increases after about 300 connections. I cannot get past 400 connections without CPU reaching 100% and load average sky-rocketing.

I would expect an m1.small to handle ~1000 outgoing 128kbps streams, taking the bandwidth of the incoming mediacaster streams into account.

How can I address this?

[HTML]

2

*

1935,80,443,554

true

24000

65000

true

100

cupertinostreaming,smoothstreaming,sanjosestreaming,dvrchunkstreaming

com.wowza.wms.http.HTTPCrossdomain

*crossdomain.xml

none

com.wowza.wms.http.HTTPClientAccessPolicy

*clientaccesspolicy.xml

none

com.app.wms.http.LiveServer

app_ctrl*

admin-basic

com.app.wms.http.Stats

stat*

none

1

*

8086

true

16000

16000

true

100

com.wowza.wms.http.streammanager.HTTPStreamManager

streammanager*

admin-digest

com.wowza.wms.http.HTTPServerInfoXML

serverinfo*

admin-digest

com.wowza.wms.http.HTTPConnectionInfo

connectioninfo*

admin-digest

com.wowza.wms.http.HTTPConnectionCountsXML

connectioncounts*

admin-digest

com.wowza.wms.transcoder.httpprovider.HTTPTranscoderThumbnail

transcoderthumbnail*

admin-digest

com.wowza.wms.http.HTTPProviderMediaList

medialist*

admin-digest

com.wowza.wms.http.HTTPServerVersion

*

none

smoothstreaming

smoothstreaming

cupertinostreaming

cupertinostreaming

sanjosestreaming

sanjosestreaming

dvrchunkstreaming

dvrchunkstreaming

60

40

2

100

5

2

250

true

65000

24000

true

100

2000

90000

500

75

true

1024000

65000

50

4096

true

65000

65000

50

4096

1

2

1

1

60000

12000

20000

0

60000

true

[/HTML]

2

Hi

Try tuning the server, the guide is here,

https://www.wowza.com/docs/how-to-do-performance-tuning

Jason

3

You should not tune a EC2 instance started with one of the pre-built AMIs from here because they are tuned automatically when the instance is started. I would undo the tuning changes that you made, or start a new instance. It’s no problem to add custom HTTPProviders, etc but do not tune.

Richard

4

Also, with very low bitrate streams you are likely to hit CPU limits before network limits. In our own testing on EC2 we used 700-800Mbps test content and hit a limit at about 150Mbps.

Richard

5

Hi Jason,

Thanks for the reply.

I’ve already followed the tuning guide. I’ve posted my VHost.xml above, it’s tuned for a single-core server (ProcessorCount 1-2, as per the tuning guide).

Unfortunately performance is still bound by CPU, rather than network. What else could be a factor?

6

yep that’s what I figured. Thanks for the reply.

7

(It’s a custom-built EC2 instance so it did need tuning, btw).

8

It seems to me you are hitting the CPU limits of the m1.small instance.

9

uniticon,

Lower bitrate streams means you can stream more streams using the same overall bandwidth. Just guessing here, but I imagine more streams means increased overhead in the internal scheduler/resource manager, therefore increased CPU overhead. Also, each stream will have a small buffer, so more streams also means more RAM overhead. And for VOD, more disk I/O.

10

The lower the bitrate, the more CPU usage? It kind of makes sense because more compression cycles are going into making stream smaller but, conversely, if you do 720p streams, shouldn’t the CPU be equally (or more) used?