Wowza 4.1 nDVR edge/origin - 'store not found for vStream:'

Hello,

This post is a bit long as the problem seemed to get complicated. I’ll try my best to walk you through the situation and how I got to this point.

Our topology looks like this:

Server1

liveorigin (primary)

-36 RTMP channels as incomming streams from encoders

liveedge (points to primary|backup)

Load Balancer Server (primary)

Load Balancer Client

Server2

liveorigin (backup)

-36 RTMP channels as incomming streams from encoders

liveedge (points to primary|backup)

Load Balancer Server (backup)

Load Balancer Client

A while back, while developing our IPTV system with Wowza 3, we noticed that sometimes, but not always, an encoder would restart, and the edge application could not play the DVR stream from the origin application. Since we have 2 load balanced edge servers, often one of the edges could play the DVR stream but the other edge couldn’t. It’s as if the edge’s connection to the origin with that stream somehow gets corrupted, and nothing short of an applcation restart fixes it. This has a side effect, however, since a Wowza service restart causes all of the encoders to restart and thus creating new dead streams for the edges. If I play the DVR stream directly from the orgin application, it works fine. If I play the stream live from any application, it works as well. It’s only from an edge that it won’t play. Here is a typical example of what the state looks like;

Encoder streaming to primary and backup live origins with a stream name of ‘21’. Playback on origins would be:

Server1

http://[server1-ip]:1935/liveorigin/21/playlist.m3u8
http://[server1-ip]:1935/liveorigin/21/playlist.m3u8&DVR

Server2

http://[server2-ip]:1935/liveorigin/21/playlist.m3u8
http://[server2-ip]:1935/liveorigin/21/playlist.m3u8&DVR

Applications ‘liveedge’ on each server point to the origins; primary first and backup second. Playback on edges is:

Server1

http://[server1-ip]:1935/liveedge/21/playlist.m3u8
http://[server1-ip]:1935/liveedge/21/playlist.m3u8&DVR

Server2

http://[server2-ip]:1935/liveedge/21/playlist.m3u8
http://[server2-ip]:1935/liveedge/21/playlist.m3u8&DVR

Now, encoder restarts (for various reasons). It starts successfully streaming again but here are the playback results;

Server1

http://[server1-ip]:1935/liveorigin/21/playlist.m3u8 - [COLOR="#006400"]Success[/COLOR]
http://[server1-ip]:1935/liveorigin/21/playlist.m3u8&DVR - [COLOR="#006400"]Success[/COLOR]

Server2

http://[server2-ip]:1935/liveorigin/21/playlist.m3u8 - [COLOR="#006400"]Success[/COLOR]
http://[server2-ip]:1935/liveorigin/21/playlist.m3u8&DVR - [COLOR="#006400"]Success[/COLOR]

Server1

http://[server1-ip]:1935/liveedge/21/playlist.m3u8 - [COLOR="#006400"]Success[/COLOR]
http://[server1-ip]:1935/liveedge/21/playlist.m3u8&DVR - [COLOR="#B22222"]Failure[/COLOR]

Server2

http://[server2-ip]:1935/liveedge/21/playlist.m3u8 - [COLOR="#006400"]Success[/COLOR]
http://[server2-ip]:1935/liveedge/21/playlist.m3u8&DVR - [COLOR="#006400"]Success[/COLOR]

After more testing and watching the logs, I realized that if I restart an encoder while watching the DVR stream, I had to wait until the edge killed the connection of that stream with the origin before attempting to watch the stream again. This takes, in total, 85sec; 25sec for the edge to kill the connection with the client, and another 60sec for the edge to kill the connection with the origin. I could replicate this exact scenerio every time by just watching a stream, restarting the encoder, and trying to connect to the stream before 85sec. As soon as I saw the edge kill the connection to the orgin in the logs (85sec), then I could start watching again. That meant, that if any of our thousands of clients requested channel 21 (stream), the timer would start over and no client could play the stream from the ‘broken’ edge. This seemed like a huge bug to me, and it seemed like this was all one big issue.

So, before investing more time in Wowza 3 and the nDVR bug, I decided to try Wowza 4 and see if the same issue was there. After testing in our development environment, it seemed like the issue was resolved. I could watch a stream, restart the encoder, and continue watching as soon as the encoder started streaming again (12-15sec). This was great news, so I moved toward getting our production servers upgraded to Wowza 4.

A little while ago we pushed Wowza 4 to our production encoders. We have been having lots of issues, for various reasons (tuning, load balancing), but this nDVR issue also came back. It seems as though the problem with not being able to play the stream back right away is fixed, but we still will get an edge with a broken link to an orgin’s stream after an encoder restart. Here is a snippet of the log with filtering on for ‘definst/21’ when I try to play a broken DVR stream from an edge:

The URL I am pointing to:

http://[server1-ip]:1935/liveedge/21/playlist.m3u8&DVR

The logs on server1 when I try to connect:

2014-10-21      15:16:39        UTC     comment server  INFO    200     -       DvrLiveRepeaterReceiver.resetStream[liveedge/_definst_/21:dvrstreamingpacketizer]       -       -   -18002.074       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -   --       -       -
2014-10-21      15:16:43        UTC     comment server  INFO    200     -       DvrLiveRepeaterReceiver.initLiveStreamPacketizer[liveedge/_definst_/21] localStreamName:21 localPackName:dvrstreamingpacketizer      -       -       -       18005.659       -       -       -       -       -       -       -       -       -       -       -       -       -       -   --       -       -       -       -       -       -       -       -       -
2014-10-21      15:16:43        UTC     comment server  INFO    200     -       DvrStreamManagerBase.initProperties[liveedge/_definst_/21] : {Properties: dvrMediaCacheEnabled: false, isDvrPacketizer: true, dvrCupertinoPlaylistAllowCaching: true}        -       -       -       18005.66        -       -       -       -       -       -       -       -       -   --       -       -       -       -       -       -       -       -       -       -       -       -       -       -
2014-10-21      15:16:43        UTC     comment server  INFO    200     -       DvrStreamManagerBase.initStorage[liveedge/_definst_/21] : storeName: isRecorder:false hasStorage:false       -       -       -       18005.661       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -   --       -       -       -       -       -       -
2014-10-21      15:16:43        UTC     comment server  INFO    200     -       MediaStreamMap.getLiveStreamPacketizer[liveedge/_definst_/21]: Create live stream packetizer: dvrstreamingpacketizer:21      -       -       -       18005.661       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -   --       -       -       -       -       -       -       -       -
2014-10-21      15:16:43        UTC     comment server  INFO    200     -       DvrStreamStoreBase.initProperties[liveedge/_definst_/21/21.0] : ChunkOriginURL=wowz://172.20.24.10:1935/liveorigin|wowz://172.20.24.11:1935/liveorigin       -       -       -       18005.662       -       -       -       -       -       -       -       -       -       -       -   --       -       -       -       -       -       -       -       -       -       -       -       -
2014-10-21      15:16:43        UTC     comment server  INFO    200     -       DvrStreamStoreBase.initProperties[liveedge/_definst_/21/21.0] : properties:[dvrWindowDuration:0, dvrChunkGroupingSeconds:600, dvrAllowableAVPacketDelta:2000, dvrResetTimePacketDelta:200, dvrPacketDeltaToNotify:200]       -       -       -       18005.662       -       -       -   --       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -
2014-10-21      15:16:43        UTC     comment server  INFO    200     -       DvrStreamManagerBase [liveedge/_definst_/21] : Timeout occurred.  Destroying DVR packetizer.    -   --       18005.751       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -   --       -       -       -
2014-10-21      15:16:43        UTC     comment server  INFO    200     -       DvrStreamManagerBase.shutdown[liveedge/_definst_/21]    -       -       -       18005.752       -   --       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -
2014-10-21      15:16:43        UTC     comment server  INFO    200     -       DvrStreamManagerBase.removeStreamStore[liveedge/_definst_/21] : stream:{DvrStreamStoreBase streamName:21.0 isLoaded:false canPlay:true canRecord:false hasEncryption:false isLive:false isRecording:false isRecordingPaused:false props:{Properties: isDvrPacketizer: true, dvrCupertinoPlaylistAllowCaching: true, dvrMediaCacheEnabled: false}].        -       -       -       18005.752       -       -       -       -       -       -       -       -       -       -   --       -       -       -       -       -       -       -       -       -       -       -       -       -
2014-10-21      15:16:43        UTC     comment server  INFO    200     -       MediaStreamMap.removeLiveStreamPacketizer[liveedge/_definst_/21]: Destroy live stream packetizer: dvrstreamingpacketizer     -       -       -       18005.753       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -   --       -       -       -       -       -       -       -       -
2014-10-21      15:16:43        UTC     comment server  INFO    200     -       DvrStreamManagerBase.initProperties[liveedge/_definst_/21] : {Properties: dvrMediaCacheEnabled: false, isDvrPacketizer: true, dvrCupertinoPlaylistAllowCaching: true}        -       -       -       18005.753       -       -       -       -       -       -       -       -       -   --       -       -       -       -       -       -       -       -       -       -       -       -       -       -
2014-10-21      15:16:43        UTC     comment server  INFO    200     -       DvrStreamManagerBase.initStorage[liveedge/_definst_/21] : storeName: isRecorder:false hasStorage:false       -       -       -       18005.754       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -   --       -       -       -       -       -       -
2014-10-21      15:16:43        UTC     comment server  INFO    200     -       MediaStreamMap.getLiveStreamPacketizer[liveedge/_definst_/21]: Create live stream packetizer: dvrstreamingpacketizer:21      -       -       -       18005.754       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -   --       -       -       -       -       -       -       -       -
2014-10-21      15:16:43        UTC     comment server  WARN    200     -       DvrStreamManagerBase.addManifestEntries[liveedge/_definst_/21] :  store not found for vStream:21.0  --       -       18005.759       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -   --       -       -       -       -
2014-10-21      15:16:43        UTC     comment server  WARN    200     -       DvrStreamManagerBase.addManifestEntries[liveedge/_definst_/21] :  store not found for vStream:21.0  --       -       18005.764       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -   --       -       -       -       -
2014-10-21      15:16:43        UTC     comment server  WARN    200     -       DvrStreamManagerBase.addManifestEntries[liveedge/_definst_/21] :  store not found for vStream:21.0  --       -       18005.769       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -   --       -       -       -       -
2014-10-21      15:16:43        UTC     comment server  WARN    200     -       DvrStreamManagerBase.addManifestEntries[liveedge/_definst_/21] :  store not found for vStream:21.0  --       -       18005.774       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -   --       -       -       -       -
2014-10-21      15:16:43        UTC     comment server  WARN    200     -       DvrStreamManagerBase.addManifestEntries[liveedge/_definst_/21] :  store not found for vStream:21.0  --       -       18005.779       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -   --       -       -       -       -
2014-10-21      15:16:43        UTC     comment server  WARN    200     -       DvrStreamManagerBase.addManifestEntries[liveedge/_definst_/21] :  store not found for vStream:21.0  --       -       18005.783       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -   --       -       -       -       -
2014-10-21      15:16:43        UTC     comment server  WARN    200     -       DvrStreamManagerBase.addManifestEntries[liveedge/_definst_/21] :  store not found for vStream:21.0  --       -       18005.788       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -   --       -       -       -       -
2014-10-21      15:16:43        UTC     comment server  WARN    200     -       DvrStreamManagerBase.addManifestEntries[liveedge/_definst_/21] :  store not found for vStream:21.0  --       -       18005.79        -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -   --       -       -       -       -
2014-10-21      15:16:43        UTC     comment server  WARN    200     -       DvrStreamManagerBase.addManifestEntries[liveedge/_definst_/21] :  store not found for vStream:21.0  --       -       18005.793       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -   --       -       -       -       -
2014-10-21      15:16:43        UTC     comment server  WARN    200     -       DvrStreamManagerBase.addManifestEntries[liveedge/_definst_/21] :  store not found for vStream:21.0  --       -       18005.796       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -   --       -       -       -       -
2014-10-21      15:16:43        UTC     comment server  WARN    200     -       DvrStreamManagerBase.addManifestEntries[liveedge/_definst_/21] :  store not found for vStream:21.0  --       -       18005.799       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -   --       -       -       -       -
2014-10-21      15:16:43        UTC     comment server  WARN    200     -       DvrStreamManagerBase.addManifestEntries[liveedge/_definst_/21] :  store not found for vStream:21.0  --       -       18005.802       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -   --       -       -       -       -
2014-10-21      15:16:43        UTC     comment server  WARN    200     -       DvrStreamManagerBase.addManifestEntries[liveedge/_definst_/21] :  store not found for vStream:21.0  --       -       18005.806       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -   --       -       -       -       -
2014-10-21      15:16:43        UTC     comment server  WARN    200     -       DvrStreamManagerBase.addManifestEntries[liveedge/_definst_/21] :  store not found for vStream:21.0  --       -       18005.809       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -   --       -       -       -       -
2014-10-21      15:16:43        UTC     comment server  WARN    200     -       DvrStreamManagerBase.addManifestEntries[liveedge/_definst_/21] :  store not found for vStream:21.0  --       -       18005.812       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -   --       -       -       -       -
2014-10-21      15:16:43        UTC     comment server  WARN    200     -       DvrStreamManagerBase.addManifestEntries[liveedge/_definst_/21] :  store not found for vStream:21.0  --       -       18005.815       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -   --       -       -       -       -
2014-10-21      15:16:43        UTC     comment server  WARN    200     -       DvrStreamManagerBase.addManifestEntries[liveedge/_definst_/21] :  store not found for vStream:21.0  --       -       18005.818       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -   --       -       -       -       -
2014-10-21      15:16:43        UTC     comment server  WARN    200     -       DvrStreamManagerBase.addManifestEntries[liveedge/_definst_/21] :  store not found for vStream:21.0  --       -       18005.821       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -   --       -       -       -       -
2014-10-21      15:16:43        UTC     comment server  WARN    200     -       DvrStreamManagerBase.addManifestEntries[liveedge/_definst_/21] :  store not found for vStream:21.0  --       -       18005.824       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -   --       -       -       -       -
2014-10-21      15:16:43        UTC     comment server  WARN    200     -       DvrStreamManagerBase.addManifestEntries[liveedge/_definst_/21] :  store not found for vStream:21.0  --       -       18005.827       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -   --       -       -       -       -
2014-10-21      15:16:43        UTC     comment server  WARN    200     -       DvrStreamManagerBase.addManifestEntries[liveedge/_definst_/21] :  store not found for vStream:21.0  --       -       18005.83        -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -   --       -       -       -       -
2014-10-21      15:16:43        UTC     comment server  WARN    200     -       DvrStreamManagerBase.addManifestEntries[liveedge/_definst_/21] :  store not found for vStream:21.0  --       -       18005.832       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -   --       -       -       -       -
2014-10-21      15:16:43        UTC     comment server  WARN    200     -       DvrStreamManagerBase.addManifestEntries[liveedge/_definst_/21] :  store not found for vStream:21.0  --       -       18005.835       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -   --       -       -       -       -
2014-10-21      15:16:43        UTC     comment server  WARN    200     -       DvrStreamManagerBase.addManifestEntries[liveedge/_definst_/21] :  store not found for vStream:21.0  --       -       18005.837       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -   --       -       -       -       -
2014-10-21      15:16:43        UTC     comment server  WARN    200     -       DvrStreamManagerBase.addManifestEntries[liveedge/_definst_/21] :  store not found for vStream:21.0  --       -       18005.84        -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -   --       -       -       -       -
2014-10-21      15:16:43        UTC     comment server  WARN    200     -       DvrStreamManagerBase.addManifestEntries[liveedge/_definst_/21] :  store not found for vStream:21.0  --       -       18005.843       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -   --       -       -       -       -
2014-10-21      15:16:43        UTC     comment server  WARN    200     -       DvrStreamManagerBase.addManifestEntries[liveedge/_definst_/21] :  store not found for vStream:21.0  --       -       18005.845       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -   --       -       -       -       -
2014-10-21      15:16:43        UTC     comment server  WARN    200     -       DvrStreamManagerBase.addManifestEntries[liveedge/_definst_/21] :  store not found for vStream:21.0  --       -       18005.848       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -   --       -       -       -       -
2014-10-21      15:16:43        UTC     comment server  WARN    200     -       DvrStreamManagerBase.addManifestEntries[liveedge/_definst_/21] :  store not found for vStream:21.0  --       -       18005.85        -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -   --       -       -       -       -
2014-10-21      15:16:43        UTC     comment server  WARN    200     -       DvrStreamManagerBase.addManifestEntries[liveedge/_definst_/21] :  store not found for vStream:21.0  --       -       18005.853       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -   --       -       -       -       -
2014-10-21      15:16:43        UTC     comment server  WARN    200     -       DvrStreamManagerBase.addManifestEntries[liveedge/_definst_/21] :  store not found for vStream:21.0  --       -       18005.855       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -   --       -       -       -       -
2014-10-21      15:16:43        UTC     comment server  WARN    200     -       DvrStreamManagerBase.addManifestEntries[liveedge/_definst_/21] :  store not found for vStream:21.0  --       -       18005.858       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -   --       -       -       -       -
2014-10-21      15:16:43        UTC     comment server  WARN    200     -       DvrStreamManagerBase.addManifestEntries[liveedge/_definst_/21] :  store not found for vStream:21.0  --       -       18005.861       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -   --       -       -       -       -
2014-10-21      15:16:43        UTC     comment server  WARN    200     -       DvrStreamManagerBase.addManifestEntries[liveedge/_definst_/21] :  store not found for vStream:21.0  --       -       18005.863       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -   --       -       -       -       -
2014-10-21      15:16:43        UTC     comment server  WARN    200     -       DvrStreamManagerBase.addManifestEntries[liveedge/_definst_/21] :  store not found for vStream:21.0  --       -       18005.866       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -   --       -       -       -       -
2014-10-21      15:16:43        UTC     comment server  WARN    200     -       DvrStreamManagerBase.addManifestEntries[liveedge/_definst_/21] :  store not found for vStream:21.0  --       -       18005.869       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -   --       -       -       -       -
2014-10-21      15:16:43        UTC     comment server  WARN    200     -       DvrStreamManagerBase.addManifestEntries[liveedge/_definst_/21] :  store not found for vStream:21.0  --       -       18005.871       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -   --       -       -       -       -
2014-10-21      15:16:43        UTC     comment server  WARN    200     -       DvrStreamManagerBase.addManifestEntries[liveedge/_definst_/21] :  store not found for vStream:21.0  --       -       18005.874       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -   --       -       -       -       -
2014-10-21      15:16:43        UTC     comment server  WARN    200     -       DvrStreamManagerBase.addManifestEntries[liveedge/_definst_/21] :  store not found for vStream:21.0  --       -       18005.877       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -   --       -       -       -       -
2014-10-21      15:16:43        UTC     comment server  WARN    200     -       DvrStreamManagerBase.addManifestEntries[liveedge/_definst_/21] :  store not found for vStream:21.0  --       -       18005.88        -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -   --       -       -       -       -
2014-10-21      15:16:43        UTC     comment server  WARN    200     -       DvrStreamManagerBase.addManifestEntries[liveedge/_definst_/21] :  store not found for vStream:21.0  --       -       18005.883       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -   --       -       -       -       -
2014-10-21      15:16:43        UTC     comment server  WARN    200     -       DvrStreamManagerBase.addManifestEntries[liveedge/_definst_/21] :  store not found for vStream:21.0  --       -       18005.886       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -   --       -       -       -       -
2014-10-21      15:16:43        UTC     comment server  WARN    200     -       DvrStreamManagerBase.addManifestEntries[liveedge/_definst_/21] :  store not found for vStream:21.0  --       -       18005.889       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -   --       -       -       -       -
2014-10-21      15:16:43        UTC     comment server  WARN    200     -       DvrStreamManagerBase.addManifestEntries[liveedge/_definst_/21] :  store not found for vStream:21.0  --       -       18005.893       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -   --       -       -       -       -
2014-10-21      15:16:43        UTC     comment server  WARN    200     -       DvrStreamManagerBase.addManifestEntries[liveedge/_definst_/21] :  store not found for vStream:21.0  --       -       18005.896       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -   --       -       -       -       -
2014-10-21      15:16:43        UTC     comment server  WARN    200     -       DvrStreamManagerBase.addManifestEntries[liveedge/_definst_/21] :  store not found for vStream:21.0  --       -       18005.899       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -   --       -       -       -       -
2014-10-21      15:16:43        UTC     comment server  WARN    200     -       DvrStreamManagerBase.addManifestEntries[liveedge/_definst_/21] :  store not found for vStream:21.0  --       -       18005.901       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -   --       -       -       -       -
2014-10-21      15:16:43        UTC     comment server  WARN    200     -       DvrStreamManagerBase.addManifestEntries[liveedge/_definst_/21] :  store not found for vStream:21.0  --       -       18005.903       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -   --       -       -       -       -
2014-10-21      15:16:43        UTC     comment server  WARN    200     -       DvrStreamManagerBase.stateChange[liveedge/_definst_/21] :  store not found for vStream:21.0     -   --       18005.904       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -   --       -       -       -
2014-10-21      15:16:43        UTC     comment server  WARN    200     -       DvrStreamManagerBase.removeManifestEntry[liveedge/_definst_/21] :  store not found for vStream:21.0 --       -       18005.904       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -   --       -       -       -       -
2014-10-21      15:16:43        UTC     comment server  WARN    200     -       DvrStreamManagerBase.addManifestEntries[liveedge/_definst_/21] :  store not found for vStream:21.0  --       -       18005.905       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -   --       -       -       -       -
2014-10-21      15:16:43        UTC     comment server  INFO    200     -       DvrStreamManagerBase [liveedge/_definst_/21] : Timeout occurred.  Destroying DVR packetizer.    -   --       18006.037       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -   --       -       -       -
2014-10-21      15:16:43        UTC     comment server  INFO    200     -       DvrStreamManagerBase.shutdown[liveedge/_definst_/21]    -       -       -       18006.038       -   --       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -
2014-10-21      15:16:43        UTC     comment server  INFO    200     -       MediaStreamMap.removeLiveStreamPacketizer[liveedge/_definst_/21]: Destroy live stream packetizer: dvrstreamingpacketizer     -       -       -       18006.038       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -   --       -       -       -       -       -       -       -       -
2014-10-21      15:16:45        UTC     comment server  INFO    200     -       DvrStreamManagerBase.initProperties[liveedge/_definst_/21] : {Properties: dvrMediaCacheEnabled: false, isDvrPacketizer: true, dvrCupertinoPlaylistAllowCaching: true}        -       -       -       18007.5 -       -       -       -       -       -       -       -       -       -   --       -       -       -       -       -       -       -       -       -       -       -       -       -
2014-10-21      15:16:45        UTC     comment server  INFO    200     -       DvrStreamManagerBase.initStorage[liveedge/_definst_/21] : storeName: isRecorder:false hasStorage:false       -       -       -       18007.501       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -   --       -       -       -       -       -       -
2014-10-21      15:16:45        UTC     comment server  INFO    200     -       MediaStreamMap.getLiveStreamPacketizer[liveedge/_definst_/21]: Create live stream packetizer: dvrstreamingpacketizer:21      -       -       -       18007.501       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -   --       -       -       -       -       -       -       -       -
2014-10-21      15:16:45        UTC     comment server  WARN    200     -       DvrStreamManagerBase.removeManifestEntry[liveedge/_definst_/21] :  store not found for vStream:21.0 --       -       18007.501       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -   --       -       -       -       -
2014-10-21      15:16:45        UTC     comment server  WARN    200     -       DvrStreamManagerBase.addManifestEntries[liveedge/_definst_/21] :  store not found for vStream:21.0  --       -       18007.502       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -   --       -       -       -       -
2014-10-21      15:16:45        UTC     comment server  INFO    200     -       DvrStreamManagerBase [liveedge/_definst_/21] : Timeout occurred.  Destroying DVR packetizer.    -   --       18007.624       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -   --       -       -       -
2014-10-21      15:16:45        UTC     comment server  INFO    200     -       DvrStreamManagerBase.shutdown[liveedge/_definst_/21]    -       -       -       18007.625       -   --       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -
2014-10-21      15:16:45        UTC     comment server  INFO    200     -       MediaStreamMap.removeLiveStreamPacketizer[liveedge/_definst_/21]: Destroy live stream packetizer: dvrstreamingpacketizer     -       -       -       18007.625       -       -       -       -       -       -       -       -       -       -       -       -       -       -       -   --       -       -       -       -       -       -       -       -

I know you’ll probably need more information, but this is a good start. If you have any ideas, please let me know. Thanks!

Jordan

Hi !

I am new on this forum and I don’t know if you’ve got an answer for that point but just to say that I have the same warning since I upgraded to the version 4.1.

I am pretty sure of my configuration and I feel like the cache system on the edge servers (nDVR + Mediacache) is a bit different than with the 4.0.6 (I’m not exactly sure of the version I had just before this last one). I don’t have enough experience on Wowza to understand what is going on but I feel like a difference since the last version… ?

Is there someone to explain if everything is normal or if it could be a bug in the last version?

Thank you,

Alex

Hello folks,

It sounds like the nDVR store may have accidentally been renamed, moved or deleted, or possibly corrupted. Double check the the DVR store is still in the correct location with the correct name.

If that all seems good you may need to open a ticket with support.

To do so please zip the following directories and send them to support@wowza.com:

[install-dir]/conf

[install-dir]/logs

[install-dir]/transcoder

[install-dir]/manager/logs

Kind regards,

Salvadore

Hi Salvadore,

Just a quick update on this. We verified what you asked us to verify, and it still doesn’t really make sense what was happening. Like I said before, whenever we would get a ‘dead’ nDVR stream, we could always playback the live version of it, and we could always playback the DVR version directly from the origin. So it appears that the edge somehow lost location of the DVR stream for that channel from the origin.

Since we ultimately want to do ABR with our 36 IPTV streams, however, and since that requires we packetize on the edge instead of the origin, we set up packetization on the edge. Since we did that, we no longer are getting corrupt nDVR files on the edge. Also, although I didn’t see anywhere in the forums on setting it up this way, we do the nDVR packetizing on the edge as well. It adds a lot of stability between origin and edge in this setup, since each edge is taking a lot more of the load. I am guessing this setup is undesirable for a couple of reasons in most scenarios because, 1) it requires an nDVR license for each edge and 2) you have constant bandwidth between origin and edge. For us, though, those issues aren’t a concern. Is there any other reason why all packetizing shouldn’t be done on the edges?

We noticed would have issues where we would get really high CPU spikes for channels that were ‘dormant’ (i.e. no one was watching them, and the first person to connect would trigger the edge requesting the stream from the origin). It would be compounded because as a client switched channels, the edge would have to establish connections to the origin very quickly. Once we kept the streams alive all the time with the packetization on the edge, this problem improved. We had to do a lot of other things for our setup to get the servers stable, but they aren’t related to this thread.

Just wanted to post in case it would help anyone!

Jordan

Hello all!

Thank you Salvador for your answer. I send my configuration to the Wowza support, I am still waiting for their answer.

I spend a bit more time on this issue and I explain a bit more here what I have:

  • On the origin server: we can create a stream and it’s saved in the DVR directory (“toto/live” for me)

  • From a live-edge-repeater: we can watch videos (so OriginUrl is OK in this side)

  • I want to use DVR + Mediacache but repeaters can’t cache chunks from a DVR-cached videos

  • So each new user on a repeater creates a new data stream between the repeater and the origin

The problem could be due from the appInstance name: we usually stream to “rtmp://server:1935/toto/live” and so the origin DVR files are saved in the “live” directory. It seems that Mediacache of repeaters try to get origin DVR files from “definst” URL/directory and not from the “/toto/live”… ? But DVR files are well saved in the “live” directory on the origin server.

To test it, I moved my origin files into the “definst” directory, and repeaters can now have access to the DVR files (and cache them by the Mediacache).

I am trying now to see if I can change my entire architecture to stream directly in the default appInstance of my “toto” application.

Does the Wowza team have any idea of the problem? I thought that my configuration was good (I did it many times) but maybe I missed something… ? Or is it a bug of the last release ? I thought it worked before with the Wowza 3.X but maybe I am wrong ?

Thank you in advance,

Alexandre