Memory panic on Windows with nDVR

After switching on 2 features:

  1. SMIL playlist with 8 streams

  2. nDVR for 10 streams for 12 hours

windows 2008 R2 64bit quad-core 4gb ram ( Java wrap 4000m memory heap size ) server enter “memory panic” mode after hour of operation and with only some connected streams

with following messages:

[HTML]STATUS | wrapper | 2012/05/17 13:06:56 | --> Wrapper Started as Service

STATUS | wrapper | 2012/05/17 13:06:56 | Launching a JVM…

ERROR | wrapper | 2012/05/17 14:55:14 | JVM appears hung: Timed out waiting for signal from JVM.

ERROR | wrapper | 2012/05/17 14:55:15 | JVM did not exit on request, terminated

STATUS | wrapper | 2012/05/17 14:55:21 | Launching a JVM…

ERROR | wrapper | 2012/05/17 17:09:36 | JVM appears hung: Timed out waiting for signal from JVM.

ERROR | wrapper | 2012/05/17 17:09:37 | JVM did not exit on request, terminated

STATUS | wrapper | 2012/05/17 17:09:43 | Launching a JVM…

ERROR | wrapper | 2012/05/17 19:33:51 | JVM appears hung: Timed out waiting for signal from JVM.

ERROR | wrapper | 2012/05/17 19:33:52 | JVM did not exit on request, terminated

STATUS | wrapper | 2012/05/17 19:33:58 | Launching a JVM…

ERROR | wrapper | 2012/05/17 19:34:30 | Startup failed: Timed out waiting for a signal from the JVM.

ERROR | wrapper | 2012/05/17 19:34:30 | JVM did not exit on request, terminated

STATUS | wrapper | 2012/05/17 19:34:35 | Launching a JVM…

STATUS | wrapper | 2012/05/17 20:21:55 | <-- Wrapper Stopped[/HTML]

Are there any extra RAM requirements to run nDVR?

Could we put in production environment?

Some extra info:

Windows memory processes with nDVR:

Processes after disabling SMIL and nDVR:

Some error messages:

[HTML]#Fields: x-severity x-category x-event date time c-client-id c-ip c-port cs-bytes sc-bytes x-duration x-sname x-stream-id x-spos sc-stream-bytes cs-stream-bytes x-file-size x-file-length x-ctx x-comment

WARN server comment 2012-05-17 19:35:00 - - - - - 5.85 - - - - - - - - QTUtils.parseAtomstbl[C:\Program Files (x86)\Wowza Media Systems\Wowza Media Server 3.0.4\content\xxxx.f4v]: Error reading ‘stsd’ section. Structure is zero size: smhd:skip

WARN server comment 2012-05-17 19:35:56 - - - - - 61.714 - - - - - - - - DvrManifestContainerParser.readContainerManifest[live/definst]: File ‘0063_10_00\dvrManifest_0063_10_00.txt’ does not exist.

ERROR server comment 2012-05-17 19:35:56 - - - - - 61.714 - - - - - - - - DvrManifestContainerParser.retrieveContainerManifest: Error reading Manifest.

WARN server comment 2012-05-17 19:36:19 - - - - - 0.453 - - - - - - - - HTTPStreamerAdapterCupertinoStreamer.getAppInstance: Stream not found [xxx/stream/playlist.m3u8]: stream

WARN server comment 2012-05-17 19:37:49 - - - - - 174.58 - - - - - - - - LiveStreamPacketizerSmoothStreaming.handlePacket[live/definst/xxx]: Fragment duration greater than suggested range of 1-4 seconds. Adjust keyframe interval accordingly: Fragment durations: [1.4,2.9,6.0]

WARN server comment 2012-05-17 19:43:41 - - - - - 526.484 - - - - - - - - LiveStreamPacketizerSmoothStreaming.handlePacket[live/definst/xxx]: Fragment duration greater than suggested range of 1-4 seconds. Adjust keyframe interval accordingly: Fragment durations: [10.0,10.0,5.1]

WARN server comment 2012-05-17 19:43:42 - - - - - 528.229 - - - - - - - - LiveStreamDvrRecorder.reportCodecInformation[live/definst/stream3][avc1.100.30]: H.264 (Video may not be playable on older iPhone and iPod touch devices where Baseline/Level 3.0 or lower is required) Video info: : {MediaCodecInfoVideo: codec:H264, profile:High, level:3.0, frameSize:640x480, displaySize:640x480, frameRate:25,000000}

WARN server comment 2012-05-17 19:43:42 - - - - - 528.229 - - - - - - - - LiveStreamDvrRecorder.reportCodecInformation[live/definst/stream3][No audio]: Smooth must have audio and video to play.

WARN server comment 2012-05-17 19:44:06 - - - - - 552.005 - - - - - - - - LiveStreamPacketizerSmoothStreaming.handlePacket[live/definst/xxx]: Fragment duration greater than suggested range of 1-4 seconds. Adjust keyframe interval accordingly: Fragment durations: [3.0,2.9,5.9]

WARN server comment 2012-05-17 19:47:40 - - - - - 765.836 - - - - - - - - LiveStreamDvrRecorder.reportCodecInformation[live/definst/xxx][avc1.66.31]: H.264 (Video may not be playable on older iPhone and iPod touch devices where Baseline/Level 3.0 or lower is required) Video info: : {MediaCodecInfoVideo: codec:H264, profile:Baseline, level:3.1, frameSize:640x480, displaySize:640x480, frameRate:25,000000}

WARN server comment 2012-05-17 19:47:40 - - - - - 765.849 - - - - - - - - LiveStreamPacketizerSmoothStreaming.handlePacket[live/definst/xxxx]: Fragment duration greater than suggested range of 1-4 seconds. Adjust keyframe interval accordingly: Fragment durations: [6.6,5.0,5.0]

WARN server comment 2012-05-17 19:47:40 - - - - - 766.19 - - - - - - - - DvrChannelManifest.remove[live/definst/xxx/xxxx.0] : attempt to remove manifest entry index:1 that does not exist. range:0-136

WARN server comment 2012-05-17 19:48:00 - - - - - 785.957 - - - - - - - - LiveStreamDvrRecorder.reportCodecInformation[live/definst/xx][avc1.77.30]: H.264 (Video may not be playable on older iPhone and iPod touch devices where Baseline/Level 3.0 or lower is required) Video info: : {MediaCodecInfoVideo: codec:H264, profile:Main, level:3.0, frameSize:640x360, displaySize:640x360, frameRate:25,000000}

WARN server comment 2012-05-17 19:49:42 - - - - - 887.509 - - - - - - - - LiveStreamPacketizerSmoothStreaming.handlePacket[live/definst/xx]: Fragment duration greater than suggested range of 1-4 seconds. Adjust keyframe interval accordingly: Fragment durations: [6.2,6.0,5.9]

WARN server comment 2012-05-17 19:49:51 - - - - - 896.393 - - - - - - - - LiveStreamDvrRecorder.reportCodecInformation[live/definst/xx][avc1.66.31]: H.264 (Video may not be playable on older iPhone and iPod touch devices where Baseline/Level 3.0 or lower is required) Video info: : {MediaCodecInfoVideo: codec:H264, profile:Baseline, level:3.1, frameSize:720x404, displaySize:720x404, frameRate:29,970000}

WARN server comment 2012-05-17 19:50:19 - - - - - 924.983 - - - - - - - - LiveStreamDvrRecorder.reportCodecInformation[live/definst/xxx][avc1.66.31]: H.264 (Video may not be playable on older iPhone and iPod touch devices where Baseline/Level 3.0 or lower is required) Video info: : {MediaCodecInfoVideo: codec:H264, profile:Baseline, level:3.1, frameSize:720x404, displaySize:720x404, frameRate:29,970000}

WARN server comment 2012-05-17 19:50:37 - - - - - 942.324 - - - - - - - - LiveStreamDvrRecorder.reportCodecInformation[live/definst/xx][avc1.100.30]: H.264 (Video may not be playable on older iPhone and iPod touch devices where Baseline/Level 3.0 or lower is required) Video info: : {MediaCodecInfoVideo: codec:H264, profile:High, level:3.0, frameSize:640x480, displaySize:640x480, frameRate:25,000000}

WARN server comment 2012-05-17 19:50:37 - - - - - 942.371 - - - - - - - - LiveStreamPacketizerSmoothStreaming.handlePacket[live/definst/xxxxx]: Fragment duration greater than suggested range of 1-4 seconds. Adjust keyframe interval accordingly: Fragment durations: [12.0,12.0,12.0]

WARN server comment 2012-05-17 19:50:37 - - - - - 942.823 - - - - - - - - DvrChannelManifest.remove[live/definst/xx/xxx.0] : attempt to remove manifest entry index:1 that does not exist. range:0-200

WARN server comment 2012-05-17 19:51:04 - - - - - 969.468 - - - - - - - - LiveStreamDvrRecorder.reportCodecInformation[live/_definst_xx_SV][avc1.77.30]: H.264 (Video may not be playable on older iPhone and iPod touch devices where Baseline/Level 3.0 or lower is required) Video info: : {MediaCodecInfoVideo: codec:H264, profile:Main, level:3.0, frameSize:720x404, displaySize:720x404, frameRate:25,000000}

WARN server comment 2012-05-17 19:51:04 - - - - - 969.492 - - - - - - - - LiveStreamPacketizerSmoothStreaming.handlePacket[live/definst/xx_SV]: Fragment duration greater than suggested range of 1-4 seconds. Adjust keyframe interval accordingly: Fragment durations: [6.2,6.0,5.9]

WARN server comment 2012-05-17 19:54:53 - - - - - 1199.18 - - - - - - - - DvrManifestContainerParser.readContainerManifest[live/definst]: File ‘0063_10_00\dvrManifest_0063_10_00.txt’ does not exist.

ERROR server comment 2012-05-17 19:54:53 - - - - - 1199.18 - - - - - - - - DvrManifestContainerParser.retrieveContainerManifest: Error reading Manifest.

WARN server comment 2012-05-17 20:00:00 - - - - - 1506.052 - - - - - - - - DvrChannelManifest.remove[live/definst/vvvv/vvvv.0] : attempt to remove manifest entry index:1 that does not exist. range:0-91

WARN server comment 2012-05-17 20:04:22 - - - - - 1767.754 - - - - - - - - LiveStreamPacketizerBase.addPacketizerEvent: Too many packets in event queue, pausing, errorCount: 0

WARN server comment 2012-05-17 20:04:22 - - - - - 1767.764 - - - - - - - - LiveStreamPacketizerBase.addPacketizerEvent: Too many packets in event queue, pausing, errorCount: 0

WARN server comment 2012-05-17 20:04:22 - - - - - 1767.767 - - - - - - - - LiveStreamPacketizerBase.addPacketizerEvent: Too many packets in event queue, pausing, errorCount: 0

WARN server comment 2012-05-17 20:04:22 - - - - - 1767.77 - - - - - - - - LiveStreamPacketizerBase.addPacketizerEvent: Too many packets in event queue, pausing, errorCount: 0

WARN server comment 2012-05-17 20:04:22 - - - - - 1767.774 - - - - - - - - LiveStreamPacketizerBase.addPacketizerEvent: Too many packets in event queue, pausing, errorCount: 0

WARN server comment 2012-05-17 20:04:22 - - - - - 1767.786 - - - - - - - - LiveStreamPacketizerBase.addPacketizerEvent: Too many packets in event queue, pausing, errorCount: 0

WARN server comment 2012-05-17 20:04:22 - - - - - 1767.795 - - - - - - - - LiveStreamPacketizerBase.addPacketizerEvent: Too many packets in event queue, pausing, errorCount: 0

WARN server comment 2012-05-17 20:04:22 - - - - - 1767.799 - - - - - - - - LiveStreamPacketizerBase.addPacketizerEvent: Too many packets in event queue, pausing, errorCount: 0

WARN server comment 2012-05-17 20:04:22 - - - - - 1767.801 - - - - - - - - LiveStreamPacketizerBase.addPacketizerEvent: Too many packets in event queue, pausing, errorCount: 0

WARN server comment 2012-05-17 20:04:22 - - - - - 1767.807 - - - - - - - - LiveStreamPacketizerBase.addPacketizerEvent: Too many packets in event queue, pausing, errorCount: 0

WARN server comment 2012-05-17 20:04:22 - - - - - 1767.812 - - - - - - - - LiveStreamPacketizerBase.addPacketizerEvent: Too many packets in event queue, pausing, errorCount: 0

WARN server comment 2012-05-17 20:04:22 - - - - - 1767.818 - - - - - - - - LiveStreamPacketizerBase.addPacketizerEvent: Too many packets in event queue, pausing, errorCount: 0

WARN server comment 2012-05-17 20:04:22 - - - - - 1767.827 - - - - - - - - LiveStreamPacketizerBase.addPacketizerEvent: Too many packets in event queue, pausing, errorCount: 0

WARN server comment 2012-05-17 20:04:22 - - - - - 1767.831 - - - - - - - - LiveStreamPacketizerBase.addPacketizerEvent: Too many packets in event queue, pausing, errorCount: 0

WARN server comment 2012-05-17 20:04:22 - - - - - 1767.832 - - - - - - - - LiveStreamPacketizerBase.addPacketizerEvent: Too many packets in event queue, pausing, errorCount: 0

WARN server comment 2012-05-17 20:04:22 - - - - - 1767.838 - - - - - - - - LiveStreamPacketizerBase.addPacketizerEvent: Too many packets in event queue, pausing, errorCount: 0

WARN server comment 2012-05-17 20:04:22 - - - - - 1767.841 - - - - - - - - LiveStreamPacketizerBase.addPacketizerEvent: Too many packets in event queue, pausing, errorCount: 0

WARN server comment 2012-05-17 20:04:22 - - - - - 1767.855 - - - - - - - - LiveStreamPacketizerBase.addPacketizerEvent: Too many packets in event queue, pausing, errorCount: 0

WARN server comment 2012-05-17 20:04:22 - - - - - 1767.861 - - - - - - - - LiveStreamPacketizerBase.addPacketizerEvent: Too many packets in event queue, pausing, errorCount: 0

WARN server comment 2012-05-17 20:04:22 - - - - - 1767.865 - - - - - - - - LiveStreamPacketizerBase.addPacketizerEvent: Too many packets in event queue, pausing, errorCount: 0

WARN server comment 2012-05-17 20:04:22 - - - - - 1767.869 - - - - - - - - LiveStreamPacketizerBase.addPacketizerEvent: Too many packets in event queue, pausing, errorCount: 0

WARN server comment 2012-05-17 20:04:22 - - - - - 1767.872 - - - - - - - - LiveStreamPacketizerBase.addPacketizerEvent: Too many packets in event queue, pausing, errorCount: 0

WARN server comment 2012-05-17 20:04:22 - - - - - 1767.874 - - - - - - - - LiveStreamPacketizerBase.addPacketizerEvent: Too many packets in event queue, pausing, errorCount: 0

WARN server comment 2012-05-17 20:04:22 - - - - - 1767.876 - - - - - - - - LiveStreamPacketizerBase.addPacketizerEvent: Too many packets in event queue, pausing, errorCount: 0

WARN server comment 2012-05-17 20:04:22 - - - - - 1767.877 - - - - - - - - LiveStreamPacketizerBase.addPacketizerEvent: Too many packets in event queue, pausing, errorCount: 0

WARN server comment 2012-05-17 20:04:22 - - - - - 1767.878 - - - - - - - - LiveStreamPacketizerBase.addPacketizerEvent: Too many packets in event queue, pausing, errorCount: 0

WARN server comment 2012-05-17 20:04:22 - - - - - 1767.88 - - - - - - - - LiveStreamPacketizerBase.addPacketizerEvent: Too many packets in event queue, pausing, errorCount: 0

WARN server comment 2012-05-17 20:08:05 - - - - - 1990.537 - - - - - - - - LiveStreamDvrRecorder.reportCodecInformation[live/definst/stream2][avc1.100.30]: H.264 (Video may not be playable on older iPhone and iPod touch devices where Baseline/Level 3.0 or lower is required) Video info: : {MediaCodecInfoVideo: codec:H264, profile:High, level:3.0, frameSize:640x480, displaySize:640x480, frameRate:25,000000}

WARN server comment 2012-05-17 20:09:48 - - - - - 2094.126 - - - - - - - - DvrStreamManagerBase.addManifestEntries[live/definst/xxx] : store not found for vStream:test_signal.0

WARN server comment 2012-05-17 20:09:48 - - - - - 2094.138 - - - - - - - - LiveStreamDvrRecorder.reportCodecInformation[live/definst/xxx][avc1.77.40]: H.264 (Video may not be playable on older iPhone and iPod touch devices where Baseline/Level 3.0 or lower is required) Video info: : {MediaCodecInfoVideo: codec:H264, profile:Main, level:4.0, frameSize:1920x1080, displaySize:1920x1080, frameRate:0,000000}

WARN server comment 2012-05-17 20:19:58 - - - - - 2703.776 - - - - - - - - LiveStreamPacketizerSmoothStreaming.handlePacket[live/definst/xxx]: Fragment duration greater than suggested range of 1-4 seconds. Adjust keyframe interval accordingly: Fragment durations: [6.0,6.0,2.8]

ERROR server comment 2012-05-17 20:21:52 - - - - - 2818.133 - - - - - - - - execute: java.util.concurrent.RejectedExecutionException: Task com.wowza.wms.stream.live.LiveStreamPacketizerRunner@2af75e5 rejected from com.wowza.wms.vhost.ThreadPoolExecutorReinit@63ec3e7[Shutting down, pool size = 90, active threads = 2, queued tasks = 0, completed tasks = 3443884]

ERROR server comment 2012-05-17 20:21:52 - - - - - 2818.133 - - - - - - - - execute: java.util.concurrent.RejectedExecutionException: Task com.wowza.wms.stream.live.LiveStreamPacketizerRunner@7421432d rejected from com.wowza.wms.vhost.ThreadPoolExecutorReinit@63ec3e7[Shutting down, pool size = 90, active threads = 2, queued tasks = 0, completed tasks = 3443884]

ERROR server comment 2012-05-17 20:21:52 - - - - - 2818.137 - - - - - - - - execute: java.util.concurrent.RejectedExecutionException: Task com.wowza.wms.stream.live.LiveStreamPacketizerRunner@6874615b rejected from com.wowza.wms.vhost.ThreadPoolExecutorReinit@63ec3e7[Shutting down, pool size = 47, active threads = 2, queued tasks = 0, completed tasks = 3443884]

ERROR server comment 2012-05-17 20:21:52 - - - - - 2818.133 - - - - - - - - execute: java.util.concurrent.RejectedExecutionException: Task com.wowza.wms.stream.live.LiveStreamPacketizerRunner@4d650ec3 rejected from com.wowza.wms.vhost.ThreadPoolExecutorReinit@63ec3e7[Shutting down, pool size = 84, active threads = 2, queued tasks = 0, completed tasks = 3443884]

ERROR server comment 2012-05-17 20:21:52 - - - - - 2818.142 - - - - - - - - execute: java.util.concurrent.RejectedExecutionException: Task com.wowza.wms.stream.live.LiveStreamPacketizerRunner@3de39c75 rejected from com.wowza.wms.vhost.ThreadPoolExecutorReinit@63ec3e7[Shutting down, pool size = 14, active threads = 2, queued tasks = 0, completed tasks = 3443884]

ERROR server comment 2012-05-17 20:21:52 - - - - - 2818.144 - - - - - - - - execute: java.util.concurrent.RejectedExecutionException: Task com.wowza.wms.stream.live.LiveStreamPacketizerRunner@4974bf81 rejected from com.wowza.wms.vhost.ThreadPoolExecutorReinit@63ec3e7[Shutting down, pool size = 2, active threads = 2, queued tasks = 0, completed tasks = 3443884]

ERROR server comment 2012-05-17 20:21:52 - - - - - 2818.142 - - - - - - - - execute: java.util.concurrent.RejectedExecutionException: Task com.wowza.wms.stream.live.LiveStreamPacketizerRunner@8d49a62 rejected from com.wowza.wms.vhost.ThreadPoolExecutorReinit@63ec3e7[Shutting down, pool size = 14, active threads = 2, queued tasks = 0, completed tasks = 3443884]

ERROR server comment 2012-05-17 20:21:52 - - - - - 2818.141 - - - - - - - - execute: java.util.concurrent.RejectedExecutionException: Task com.wowza.wms.stream.live.LiveStreamDvrRunner@30a3d2f0 rejected from com.wowza.wms.vhost.ThreadPoolExecutorReinit@63ec3e7[Shutting down, pool size = 37, active threads = 2, queued tasks = 0, completed tasks = 3443884]

ERROR server comment 2012-05-17 20:21:52 - - - - - 2818.136 - - - - - - - - execute: java.util.concurrent.RejectedExecutionException: Task com.wowza.wms.stream.live.LiveStreamPacketizerRunner@45427824 rejected from com.wowza.wms.vhost.ThreadPoolExecutorReinit@63ec3e7[Shutting down, pool size = 63, active threads = 2, queued tasks = 0, completed tasks = 3443884]

ERROR server comment 2012-05-17 20:21:52 - - - - - 2818.145 - - - - - - - - execute: java.util.concurrent.RejectedExecutionException: Task com.wowza.wms.stream.live.LiveStreamDvrRunner@3158f815 rejected from com.wowza.wms.vhost.ThreadPoolExecutorReinit@63ec3e7[Shutting down, pool size = 2, active threads = 2, queued tasks = 0, completed tasks = 3443884]

ERROR server comment 2012-05-17 20:21:52 - - - - - 2818.144 - - - - - - - - execute: java.util.concurrent.RejectedExecutionException: Task com.wowza.wms.stream.live.LiveStreamDvrRunner@49cb0b64 rejected from com.wowza.wms.vhost.ThreadPoolExecutorReinit@63ec3e7[Shutting down, pool size = 2, active threads = 2, queued tasks = 0, completed tasks = 3443884]

ERROR server comment 2012-05-17 20:21:52 - - - - - 2818.144 - - - - - - - - execute: java.util.concurrent.RejectedExecutionException: Task com.wowza.wms.stream.live.LiveStreamDvrRunner@fc8d3d4 rejected from com.wowza.wms.vhost.ThreadPoolExecutorReinit@63ec3e7[Shutting down, pool size = 2, active threads = 2, queued tasks = 0, completed tasks = 3443884]

ERROR server comment 2012-05-17 20:21:52 - - - - - 2818.143 - - - - - - - - execute: java.util.concurrent.RejectedExecutionException: Task com.wowza.wms.stream.live.LiveStreamDvrRunner@42c2e33c rejected from com.wowza.wms.vhost.ThreadPoolExecutorReinit@63ec3e7[Shutting down, pool size = 2, active threads = 2, queued tasks = 0, completed tasks = 3443884]

ERROR server comment 2012-05-17 20:21:52 - - - - - 2818.143 - - - - - - - - execute: java.util.concurrent.RejectedExecutionException: Task com.wowza.wms.stream.live.LiveStreamDvrRunner@76aad4bc rejected from com.wowza.wms.vhost.ThreadPoolExecutorReinit@63ec3e7[Shutting down, pool size = 8, active threads = 2, queued tasks = 0, completed tasks = 3443884]

ERROR server comment 2012-05-17 20:21:52 - - - - - 2818.143 - - - - - - - - execute: java.util.concurrent.RejectedExecutionException: Task com.wowza.wms.stream.live.LiveStreamDvrRunner@52ab3e03 rejected from com.wowza.wms.vhost.ThreadPoolExecutorReinit@63ec3e7[Shutting down, pool size = 12, active threads = 2, queued tasks = 0, completed tasks = 3443884]

ERROR server comment 2012-05-17 20:21:52 - - - - - 2818.166 - - - - - - - - execute: java.util.concurrent.RejectedExecutionException: Task com.wowza.wms.stream.live.LiveStreamPacketizerRunner@7764efb2 rejected from com.wowza.wms.vhost.ThreadPoolExecutorReinit@63ec3e7[Shutting down, pool size = 2, active threads = 2, queued tasks = 0, completed tasks = 3443884]

ERROR server comment 2012-05-17 20:21:52 - - - - - 2818.225 - - - - - - - - execute: java.util.concurrent.RejectedExecutionException: Task com.wowza.wms.stream.live.LiveStreamPacketizerRunner@52d46c43 rejected from com.wowza.wms.vhost.ThreadPoolExecutorReinit@63ec3e7[Shutting down, pool size = 2, active threads = 2, queued tasks = 0, completed tasks = 3443884]

ERROR server comment 2012-05-17 20:21:52 - - - - - 2818.154 - - - - - - - - execute: java.util.concurrent.RejectedExecutionException: Task com.wowza.wms.stream.live.LiveStreamPacketizerRunner@5502043f rejected from com.wowza.wms.vhost.ThreadPoolExecutorReinit@63ec3e7[Shutting down, pool size = 2, active threads = 2, queued tasks = 0, completed tasks = 3443884]

ERROR server comment 2012-05-17 20:21:52 - - - - - 2818.145 - - - - - - - - execute: java.util.concurrent.RejectedExecutionException: Task com.wowza.wms.stream.live.LiveStreamDvrRunner@48074402 rejected from com.wowza.wms.vhost.ThreadPoolExecutorReinit@63ec3e7[Shutting down, pool size = 2, active threads = 2, queued tasks = 0, completed tasks = 3443884]

ERROR server comment 2012-05-17 20:21:52 - - - - - 2818.226 - - - - - - - - execute: java.util.concurrent.RejectedExecutionException: Task com.wowza.wms.stream.live.LiveStreamDvrRunner@3147f58a rejected from com.wowza.wms.vhost.ThreadPoolExecutorReinit@63ec3e7[Shutting down, pool size = 2, active threads = 2, queued tasks = 0, completed tasks = 3443884]

ERROR server comment 2012-05-17 20:21:52 - - - - - 2818.226 - - - - - - - - execute: java.util.concurrent.RejectedExecutionException: Task com.wowza.wms.stream.live.LiveStreamDvrRunner@556847e4 rejected from com.wowza.wms.vhost.ThreadPoolExecutorReinit@63ec3e7[Shutting down, pool size = 2, active threads = 2, queued tasks = 0, completed tasks = 3443884]

ERROR server comment 2012-05-17 20:21:52 - - - - - 2818.225 - - - - - - - - execute: java.util.concurrent.RejectedExecutionException: Task com.wowza.wms.stream.live.LiveStreamDvrRunner@780fcd30 rejected from com.wowza.wms.vhost.ThreadPoolExecutorReinit@63ec3e7[Shutting down, pool size = 2, active threads = 2, queued tasks = 0, completed tasks = 3443884]

#Version: 1.0[/HTML]

Wrapper setup:

[HTML]@echo off

set _EXECJAVA=java

set JAVA_OPTS=-server -Xmx4000M

rem If running JDK, uncomment to run server environment (faster)

rem set JAVA_OPTS=%JAVA_OPTS% -server

rem Better garbage collection setting to avoid long pauses

set JAVA_OPTS=%JAVA_OPTS% -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:+UseParNewGC

rem Uncomment to fix multicast crosstalk problem when streams share multicast port

rem set JAVA_OPTS=%JAVA_OPTS% -Djava.net.preferIPv4Stack=true

rem Uncomment to force Java to use specific language settings

rem set JAVA_OPTS=%JAVA_OPTS% -Duser.language=en -Duser.country=US

set EXECCSCRIPT=cscript

if not exist “%SystemRoot%\SysWOW64\cscript.exe” goto skip64bitcscript

set EXECCSCRIPT="%SystemRoot%\SysWOW64\cscript.exe"

:skip64bitcscript

set WMSCONFIG_URL=

rem set WMSAPP_HOME=C:/wms2

rem set WMSCONFIG_HOME=C:/wms2

set WMSENVOK=“false”

rem Guess WMSAPP_HOME if not defined

set CURRENT_DIR=%cd%

if not “%WMSAPP_HOME%” == “” goto gotAppHome

set WMSAPP_HOME=%CURRENT_DIR%

if exist “%WMSAPP_HOME%\bin\startup.bat” goto okAppHome

cd …

set WMSAPP_HOME=%cd%

cd %CURRENT_DIR%

:gotAppHome

if exist “%WMSAPP_HOME%\bin\startup.bat” goto okAppHome

%EXECSCRIPT% “displaymsg.vbs” “The WMSAPP_HOME environment variable is missing or not defined correctly (%WMSAPP_HOME%). This environment variable is needed to run the server.”

goto end

:okAppHome

rem Guess WMSCONFIG_HOME if not defined

set CURRENT_DIR=%cd%

if not “%WMSCONFIG_HOME%” == “” goto gotConfigHome

set WMSCONFIG_HOME=%WMSAPP_HOME%

if exist “%WMSCONFIG_HOME%\conf\Server.license” goto okConfigHome

cd …

set WMSCONFIG_HOME=%cd%

cd %CURRENT_DIR%

:gotConfigHome

if exist “%WMSCONFIG_HOME%\conf\Server.license” goto okConfigHome

%EXECSCRIPT% “displaymsg.vbs” “The WMSCONFIG_HOME environment variable is missing or not defined correctly (%WMSCONFIG_HOME%). This environment variable is needed to run the server.”

goto end

:okConfigHome

%EXECSCRIPT% “checkjavahome.vbs” //Nologo

if not %errorlevel% == 0 goto end

set WMSENVOK=“true”

set _EXECJAVA=%JAVA_HOME%\bin\java.exe

:end[/HTML]

and

[HTML]#********************************************************************

Wrapper Properties

#********************************************************************

Java Application

wrapper.java.command=%JAVA_HOME%\bin\java.exe

Java Main class. This class must implement the WrapperListener interface

or guarantee that the WrapperManager class is initialized. Helper

classes are provided to do this for you. See the Integration section

of the documentation for details.

wrapper.java.mainclass=org.tanukisoftware.wrapper.WrapperStartStopApp

Java Classpath (include wrapper.jar) Add class path elements as

needed starting from 1

wrapper.java.classpath.1=./wms-bootstrap.jar

wrapper.java.classpath.2=./wrapper.jar

Java Library Path (location of Wrapper.DLL or libwrapper.so)

wrapper.java.library.path.1=./

NOTE: Here you can configure the JVM’s built in JMX interface.

See the “Server Management Console and Monitoring” chapter

of the “User’s Guide” for more information on how to configure the

remote JMX interface in the [install-dir]/conf/Server.xml file.

Java Additional Parameters

Make sure the uncommented items are numbered sequentially

wrapper.java.additional.1=-Xmx4000M

wrapper.java.additional.2=-Dcom.wowza.wms.native.base=“win”

wrapper.java.additional.3=-Dcom.wowza.wms.runmode=“service”

wrapper.java.additional.4=-Dcom.wowza.wms.ConfigURL=

wrapper.java.additional.5=-Dcom.sun.management.jmxremote=true

wrapper.java.additional.6=-Dorg.tanukisoftware.wrapper.WrapperManager.mbean=true

wrapper.java.additional.7=-Dorg.tanukisoftware.wrapper.WrapperManager.mbean.testing=false

wrapper.java.additional.8=-Djava.net.preferIPv4Stack=true

If running JDK, uncomment to run server environment (faster)

wrapper.java.additional.9=-server

Uncomment to force Java to use specific language settings

#wrapper.java.additional.10=-Duser.language=en

#wrapper.java.additional.11=-Duser.country=US

Initial Java Heap Size (in MB)

#wrapper.java.initmemory=3

Maximum Java Heap Size (in MB)

Memory setting has been moved above to accomodate larger heap sizes

#[do not use - see wrapper.java.additional.1 property] wrapper.java.maxmemory=768

Application parameters. Add parameters as needed starting from 1

wrapper.app.parameter.1=com.wowza.wms.bootstrap.Bootstrap

wrapper.app.parameter.2=1

wrapper.app.parameter.3=start

wrapper.app.parameter.4=com.wowza.wms.bootstrap.Bootstrap

wrapper.app.parameter.5=true

wrapper.app.parameter.6=1

wrapper.app.parameter.7=stop

#********************************************************************

Wrapper Logging Properties

#********************************************************************

Format of output for the console. (See docs for formats)

wrapper.console.format=PM

Log Level for console output. (See docs for log levels)

wrapper.console.loglevel=INFO

Log file to use for wrapper output logging.

wrapper.logfile=…/logs/wowzamediaserver_service.log

Format of output for the log file. (See docs for formats)

wrapper.logfile.format=LPTM

Log Level for log file output. (See docs for log levels)

wrapper.logfile.loglevel=STATUS

Maximum size that the log file will be allowed to grow to before

the log is rolled. Size is specified in bytes. The default value

of 0, disables log rolling. May abbreviate with the ‘k’ (kb) or

‘m’ (mb) suffix. For example: 10m = 10 megabytes.

wrapper.logfile.maxsize=0

Maximum number of rolled log files which will be allowed before old

files are deleted. The default value of 0 implies no limit.

wrapper.logfile.maxfiles=0

Log Level for sys/event log output. (See docs for log levels)

wrapper.syslog.loglevel=NONE

#********************************************************************

Wrapper Windows Properties

#********************************************************************

Title to use when running as a console

wrapper.console.title=Wowza Media Server

#********************************************************************

Wrapper Windows NT/2000/XP Service Properties

#********************************************************************

WARNING - Do not modify any of these properties when an application

using this configuration file has been installed as a service.

Please uninstall the service before modifying this section. The

service can then be reinstalled.

Name of the service

wrapper.ntservice.name=WowzaMediaServer

Display name of the service

wrapper.ntservice.displayname=Wowza Media Server

Description of the service

wrapper.ntservice.description=Wowza Media Server

Service dependencies. Add dependencies as needed starting from 1

#wrapper.ntservice.dependency.1=

Mode in which the service is installed. AUTO_START or DEMAND_START

wrapper.ntservice.starttype=DEMAND_START

Allow the service to interact with the desktop.

wrapper.ntservice.interactive=true[/HTML]

With 4GB of RAM you need to set the heap size to no more than 3GB, leaving 1G for other processes/OS.

Follow the tuning guide, and re-test. If you still have the problem follow the instructions here to submit a support request. Include a link to this thread.

Take a look at these suggestions

https://www.wowza.com/docs/how-to-encode-video-on-demand-content

btw, it’s basically the same crew answering forum posts and support tickets.

Richard

Also, try this patch:

https://www.wowza.com/downloads/WowzaMediaServer-3-1-1/WowzaMediaServer3.1.1-patch8.zip

And be sure you are properly tuned

If still a problem, copy the problem file to a web server and send a link to download to support@wowza.com. Include a link to this thread for reference

Richard

Wowza should not crash on startup just because the heap is 3000. You’ll want to get that working first, since that is probably the issue we’re trying to fix. Try rebooting.

I notice you’re using high profile on the 480p stream and main profile on the 1080p stream. Shouldn’t that be the other way around?

For H.264 there is a concept of profiles. High is for HD streams, Main is for SD streams. Wherever you are setting it, either in the transcoder template, or on your encoder, is where it should be set correctly.

Moved to ticket #29210.

Actually I tried before to decrease heap memory to 3000, but there was imidiate memory panic and streaming service even did not start

Sure, Wowza service starts, but memory process go mad.

Seems like it try to initialize all streams.

Some streams from playlist starts, some not. All streams experienced huge delay or packet drops.

About profiles. Which profile settings do you mean?

If transcoder templates, we use default 2 files included in latest wowza distribution

We located the problem. Actually it is smil playlist and .mp4 file playing , not nDVR related problem.

We discovered that playing certain mp4 files in smil playlists causes overuse of memory, and if there are severel simultanious channels playing “bad” mp4 files by smil playlist server run out of memory.

We try to locate “bad” mp4 patterns.

Files recorded by wowza seems OK. h.264 and AAC

Other files are seems the same, but causing problem. We do not see any huge difference in properties using Gspot anf ffmpeg.

One of the assumptions, that Wowza do not like fragmanted mp4 files.

So we try to re-code potentially bad files using ffmpeg -i badfile.mp4 newfile.mp4 , as result we get smaller file, but still the same “memory panic” when Wowza start to play it

ANy suggestions? Do we need transcode all mp4 before putting in Wowza Playlist. Are there any tools / recommended settings?