Right, you have to deal with the OutOfMemory situation. You either need to tune, or add ram, or add server(s)
https://www.wowza.com/docs/how-to-do-performance-tuning
Wowza recommends a late model quad core cpu, 4 - 8 gigs of ram, 1 - 4gbs nic, and 64bit OS and 64bit Java JDK.
There is a load test tool you can get by sending request to test@wowza.com. There is a document to sign and return.
Richard