Hi, today one of our Wowza instance crashed with the following error
Unexpected exception.: java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached"
Wowza is running as root (setting its own ulimit to 64000) on Ubuntu 18.04
root@localhost:~# cat /proc/sys/kernel/threads-max
127824
Not sure what version you are on, but I would move to latest Beta.
I would also read my error logs to see is there are clues if maybe some other problem possibly led to the OOM/thread issue.
Scott
There can be many things that cause this and I concur @ScottKellicker’s recommendation to check the logs. Check Linux logs under /var/logs too; sometimes there’s useful info about why a process crashed.
Two (of many) possible causes that jump to my mind now:
Are you using any custom modules? In that case, check if these modules have proper resources handling. Resources / objects / connections that aren’t closed and cleaned properly may cause memory leaks and eventually exhaust the Java Heap
Do you by any chance either have custom settings for Cupertino and MPEG-DASH packetizers, e.g. very high ChunkDurationTarget or a high MaxChunkCount, and that typically combined with dozens or maybe even hundreds of streams? These packetizers do store the stream chunks/segments in memory, so if your configuration isn’t properly tuned, you may run out of memory too.
Thank you for your suggestions, we are on v4.8.5
Linux logs didn’t reveal much and Wowza logs - > everything was fine until those error and then it crashed.
It was running 1 WebRTC stream (very low quality) with about 150 viewers. The server is 8cores/16GB RAM Ubuntu 18.04