WowzaStreamingEngine Service exited on its own?

Wowza Streaming Engine
1

Hello.

Had two cases of WowzaStreamingEngine apparently exiting on its own. (Or crashing, but with exit code 0?). This has happened twice now, two weeks apart.

  • VM running Centos7
  • Wowza Streaming Engine Version 4.7.2 (build 20903)
  • Runs a continuous scheduled stream (StreamPublisher module) as well as VoD
  • Does transcoding

This is what systemctl says

$ systemctl status WowzaStreamingEngine -l
● WowzaStreamingEngine.service - WowzaStreamingEngine
   Loaded: loaded (/usr/lib/systemd/system/WowzaStreamingEngine.service; enabled; vendor preset: disabled)
   Active: active (exited) since Thu 2017-08-24 13:48:43 CEST; 19h ago
  Process: 1360 ExecStart=/usr/local/WowzaStreamingEngine/bin/systemd.sh start (code=exited, status=0/SUCCESS)
 Main PID: 1360 (code=exited, status=0/SUCCESS)

Aug 24 13:48:43 vm203.xx.xx systemd[1]: Started WowzaStreamingEngine.
Aug 24 13:48:43 vm203.xx.xx systemd[1]: Starting WowzaStreamingEngine...
Aug 24 13:48:44 vm203.xx.xx systemd.sh[1360]: /usr/local/WowzaStreamingEngine/bin/wms.sh: line 41: ${WMSPIDFILE} : ambiguous redirect
Aug 25 04:19:58 vm203.xx.xx systemd.sh[1360]: /usr/local/WowzaStreamingEngine/bin/wms.sh: line 42:  1535 Killed $_EXECJAVA $WMSTUNE_OPTS $JMXOPTIONS -Dcom.wowza.wms.runmode="service" -Dcom.wowza.wms.native.base="linux" -Dcom.wowza.wms.AppHome="$WMSAPP_HOME" -Dcom.wowza.wms.ConfigURL="$WMSCONFIG_URL" -Dcom.wowza.wms.ConfigHome="$WMSCONFIG_HOME" -cp $WMSAPP_HOME/bin/wms-bootstrap.jar com.wowza.wms.bootstrap.Bootstrap $WMSCOMMAND > /dev/null 2>&1

The Wowza error log tails warnings:

WARN	server	comment	2017-09-11	19:40:21	-	-	-	-	-	33916.255	-	-	-	-	-	-	-	-	MPEGDashWriterHandler.endChunk[liveTV2/_definst_/return2]: Stream start time calculation is off by 6438ms from original value.  Changing stream start date from "Mon Sep 11 10:15:22 CEST 2017" to "Mon Sep 11 10:15:29 CEST 2017"
WARN	server	comment	2017-09-11	19:40:23	-	-	-	-	-	33918.2	-	-	-	-	-	-	-	-	MPEGDashWriterHandler.endChunk[liveTV2/_definst_/liveTVfiller2_overlay]: Stream start time calculation is off by 7311ms from original value.  Changing stream start date from "Mon Sep 11 10:16:25 CEST 2017" to "Mon Sep 11 10:16:32 CEST 2017"
WARN	server	comment	2017-09-11	19:40:23	-	-	-	-	-	33918.2	-	-	-	-	-	-	-	-	MPEGDashWriterHandler.endChunk[liveTV2/_definst_/liveTVstream2]: Stream start time calculation is off by 7307ms from original value.  Changing stream start date from "Mon Sep 11 10:16:25 CEST 2017" to "Mon Sep 11 10:16:33 CEST 2017"
WARN	server	comment	2017-09-11	19:40:24	-	-	-	-	-	33919.026	-	-	-	-	-	-	-	-	MPEGDashWriterHandler.endChunk[liveTV2/_definst_/liveTVstream2_720p]: Stream start time calculation is off by 5080ms from original value.  Changing stream start date from "Mon Sep 11 10:16:28 CEST 2017" to "Mon Sep 11 10:16:33 CEST 2017"
WARN	server	comment	2017-09-11	19:40:34	-	-	-	-	-	33929.855	-	-	-	-	-	-	-	-	MPEGDashWriterHandler.endChunk[liveTV2/_definst_/return2]: Stream start time calculation is off by 6401ms from original value.  Changing stream start date from "Mon Sep 11 10:15:29 CEST 2017" to "Mon Sep 11 10:15:22 CEST 2017"
WARN	server	comment	2017-09-11	19:40:38	-	-	-	-	-	33933.568	-	-	-	-	-	-	-	-	MPEGDashWriterHandler.endChunk[liveTV2/_definst_/liveTVstream2]: Stream start time calculation is off by 5342ms from original value.  Changing stream start date from "Mon Sep 11 10:16:33 CEST 2017" to "Mon Sep 11 10:16:27 CEST 2017"
WARN	server	comment	2017-09-11	19:40:48	-	-	-	-	-	33943.345	-	-	-	-	-	-	-	-	MPEGDashWriterHandler.endChunk[liveTV2/_definst_/liveTVfiller2_overlay]: Stream start time calculation is off by 5324ms from original value.  Changing stream start date from "Mon Sep 11 10:16:32 CEST 2017" to "Mon Sep 11 10:16:27 CEST 2017"
WARN	server	comment	2017-09-11	19:40:50	-	-	-	-	-	33945.597	-	-	-	-	-	-	-	-	MPEGDashWriterHandler.endChunk[liveTV2/_definst_/liveTVstream2_720p]: Stream start time calculation is off by 5258ms from original value.  Changing stream start date from "Mon Sep 11 10:16:33 CEST 2017" to "Mon Sep 11 10:16:28 CEST 2017"

I’m new to these warnings. They are basically for all the streams on the server. They start to appear in log file some 24 hours before the apparent crash. Went back to previous crashes and saw same warnings starting to appear the day before. Not sure if its related though.

Thanks in advance to any help or pointers :slight_smile:

2

I have observed the same thing. Just after upgrading to 4.7.2.02 (build 21159) from 4.7.1

3

@Tobias Dedenroth Did you also get any system logs like I did:

java invoked oom-killer: gfp_mask=0x280da, order=0, oom_score_adj=0

Out of memory: Kill process 30152 (java) score 951 or sacrifice child [596713.192005] Killed process 30152 (java) total-vm:71671752kB, anon-rss:64688000kB, file-rss:0kB

Although this doesn’t make sense because my server has 64GB of RAM, and only doing packaging on a few streams. This wasn’t occuring on 4.7.1.

5

I having the same problem, my version is 4.7.5

6

I having the same problem, my version is 4.7.7

7

I’m having the same problem after trying to downgrade from 4.7.4 to 4.7.7

8

Did anybody every figure this out? I’m having the same issue on a brand new server.

9

For this group, this will be need to be looked into by Wowza Support as there could be multiple reasons behind a service crash.

Support ticket can be created here: https://www.wowza.com/support/open-ticket

A heap dump for when the service crashes to find out what could be getting stuck:
https://www.wowza.com/docs/how-to-create-a-java-heap-dump-on-wowza-streaming-engine#create-a-java-heap-dump-using-jcmd

*It’s important to take this when the issue occurs and not after restarting the Wowza service again.

Wowza conf and logs will be needed as well to get a full understanding of the issue:
[install-dir]/conf
[install-dir]/logs