Yesterday we upgraded from 3.0.5 to 3.1.1. This seemed to go fine, until I noticed Wowza started overwriting old DVR recordings for some (not all) streams (window size for all is 2592000; 30 days):
May 1 00:36 0000_00_00
May 1 00:46 0000_10_00
May 1 00:50 0000_20_00
Apr 12 09:18 0000_30_00
Apr 12 09:28 0000_40_00
Apr 12 09:38 0000_50_00
Also the server started using 100% CPU for a couple of minutes. At one time during the 100% CPU use I restarted the server to make a small configuration change that I forgot to make earlier. Might this have something to do with the corruption?
Also we’re seeing a LOT of these in the error log:
LiveStreamDvrRecorder.endChunk[livedvr/definst/oODzpwg2bQ]: Recalculating duration by estimating. Was 4153. Is now 3827.
In our use case the source sometimes drops out (once every few hours it drops out for 30 secs or so; it’s a low bitrate IPcam without audio in this particular case; think 160 kbit/s). Some extra info about this case:
true
true
false
600
-1
-1
Also, I notifed that IF one DVR-stream would get corrupted, all kinds of unpredictable things will happen and even the live feed won’t play anymore. No errors are logged if this happens.
So, my questions:
Do you have any tips, do’s/don’ts to prevent DVR archive corruption?
And are you planning on extra measures to stabilize the way Wowza handles restarts/DVR in general? It seems to be very easy to corrupt the DVR archive with a simple restart or especially a wowza upgrade. I hope you can make the DVR addon more robust as we plan on running a couple 100 of these low bitrate streams per server.
Recordings with 2592000 window size and 600 chunk grouping means Wowza will create 4320 directories per DVR per stream, or 432000 directories for 100 streams(!). Would it be wise to increase chunk grouping? What are the pros and cons?
I’m not sure if that caused the corruption. Were the streams that exhibited the problem the streams that were currently recording when the system was at 100% CPU and re-started? If you have ArchiveStrategy set to: delete and the stream is disconnected and then re-started, Wowza nDVR will delete the previous recording and record a new recording. Make sure you Server is properly tuned. The Tuning Guide is at this link.
Those log messages indicate that there is a video/audio alignment issue. Wowza nDVR relies on the incoming live video and audio being aligned. If these messages occur a couple of times when Wowza nDVR first starts up as it is buffering its initial packets, you can ignore it. If it persists and you also see skipping chunk messages, then there is a video/audio alignment issue. When there are alignment problems, chunks cannot be recorded and degrades the quality of the recording. Alignment issues should be fixed upstream from nDVR, usually fixed in the encoding process. There are nDVR AddOn properties you can try to compensate for this problem. Note that these properties will not fix the alignment problem, but may help to work around the problem. See dvrPacketSortTime and dvrAllowableAVPacketDelta in the Wowza nDVR Advanced Configuration article.
By default Wowza nDVR will wait 300,000 ms (5 minutes) for packets until it stops recording. This timeout is meant to account for when encoders/cameras disconnect and then re-start. The value should be a non-zero value or else your recording will stop if at any time there is no data. This should amply account for the 30 seconds your source drops.
Take a look at the Best Practices section of the nDVR Quick Start Guide for tips on recording length, keyframe size. That large number of directories/files will likely cause an issue with the file system handling such a large number of files. Regarding recording length, the playlist will be repeatedly requested every couple of seconds (based on key frame size), therefore an extremely large playlist size will be problematic for the player and should be avoided. See the How to use Wowza nDVR AddOn Playlist Request API article for more information on how to request a portion of your recording for play back.
This property will generate DVR chunks at least this size. They will still be broken on keyframes.
Having larger chunks for DVR has a trade-off: HTTP streaming requires 3 chunks before playback starts so 10 second chunk size will have more latency than 2 seconds chunks.
I will try to address the alignment problem later at the source. This is a separate problem.
ArchiveStrategy is set to ‘append’. And it is possible streams were recording while the server was at 100% and re-started (our application autostarts recording as soon as the input stream connects). It would be very nice if Wowza could somehow recognize it if the last chunk is corrupt and discard that and append at the last non-corrupt chunk. That by itself would probably resolve a lot of our problems. It would mean there is a visible and audible gap within the recording, but this is acceptable.
Also the Quick Start Guide mentions that nDVR has a limit of 30 hours. I do not remember reading about this limit before. Our case requires a recording duration of 720 hours (however playback size may be much smaller, e.g. 1 hour per request).
The server is already tuned (pretty low load) but to be honest nDVR isn’t. Would it be wise in our case to increase dvrChunkGroupingSeconds to e.g. 1800? Or even 3600? This would mean it would create a maximum of 24 directories per day per stream, and 6 times less manifests, so probably quicker read times? It wouldn’t fix the fact that Wowza creates a new 10 kB file every few seconds though, which may still become a problem.
According to my (double-checked) calculations 100 streams (just 16 Mbit/s!) of 720 hours would generate a whopping 120,000,000 files. Doesn’t really seem scalable. Have you got any tips/ideas/comments about this and about how we can scale? The hardware can probable handle recording 2000+ streams simultaneously, but this would mean billions(!) of files.
Ok, a quick followup question: is it possible to change the chunk size for a video or audio/video stream? The Advanced Configuration Guide only shows an example for audio only streams.
If it is not possible to change this I would like to make a feature request, namely that it would be nice if it were possible to set a target size in seconds for Wowza to aim for as chunk size. Requested behavior:
If chunk size is not set, you use keyframes to determine chunk size (= probably highest performance in high bitrate streams)
If chunk size in application config is set (e.g. to 60), Wowza attempts to store 60 seconds of content per file before moving to the next file; if it stored 60 seconds it will start a new chunk as soon as it detects a new keyframe (so if you aim for 60, you might get 62, which is no problem). Alsof if the stream gets interrupted/restarts it can start a new chunk. If I want to start DVR at a certain startpoint Wowza should be able to start at exactly that point (which means it will have to seek within the chunk to start at the correct requested keyframe). This will greatly benefit I/O performance, at the cost of a bit of CPU performance (because it will have to seek to the next best keyframe within a chunk).
Also it would be nice if (like I asked before) Wowza could be a little more resistant to corruption. E.g. remove the last (few) chunk(s) if they seem to be corrupt, before continuing recording. Also a fix script that can fix corruption would be nice (in our case with 30 days of recording it’s rather relevant that we avoid any corruption). Wanted behavior of that script is removing all corrupt chunks and making sure the manifest is correct so that all non-corrupt material will play.
Does this work if the incoming stream is RTMP instead of Flash HTTP?
And what happens if the encoder drops a frame (or similar problem)? Is it possible that the key frame frequency and chunk contents will go out of sync, causing all kinds of unpredictable behavior? Or will Wowza be able to correct these kinds of problems itself?
Ok, we will try that soon, thanks for the input, sounds promising!
Are you considering a storage method that allows appending chunks to larger files? E.g. a 10 minute file that contains 2 second chunks? A manifest-type file could contain information about the contents of the larger file. Would be more efficient, especially in a use case like ours!
Main problem at the moment seems to be that it stops recording after 108000 seconds (which is 30 hours). Even though WindowDuration is set to 2592000 (= 30 days) and though ArchiveStrategy is set to ‘append’, which seems like it should auto-delete the oldest files if the maximum duration is reached. However the advanced configuration guide and the quick-start guide make no mention of exactly what it does to the oldest files.
Logs state the following error message: (Maximum recording length of 108000 seconds exceeded. Recording stopped.)
Is there no way to record more than 30 hours anymore? Because that would be a serious problem for us!
I’ve also got a 5 GB archive that has become corrupt after the maximum duration was reached. Would it be useful for you to research this, and if so: can I send a link to the zipfiles somewhere?
Also I’d like to know if the WindowDuration can be increased back to 30 days which it was earlier. This is really important for us. I’d hate to have to roll back to an older Wowza version!
The problem with this is that the entire business case of our customer was based on a 30 day WindowDuration! This seemed to work when we tested (in 3.0.5 I believe). Now you disabled this and removed the functionality that we need which is a huge letdown as we were in the finishing phase of the project!
You mention network problems can put a whole recording on jeopardy? That could be the reason it happened to some of our recordings; the IPcam feed isn’t always that stable. This is something Wowza should be able to deal with though, imo. Network problems can also occur sometimes in very stable professional environments and Wowza should just discard corrupt packets and ignore them, or at least have the option to configure it to deal with corrupt packets like this. I am confident that you will be able to implement this and improve this in the future, paving the way to using huge window duration sizes without worries
Basically we’d have to redesign big parts of the project if we weren’t able to use Wowza DVR (we’d have to create our own chunks, map start times/dates to files), so I hope you have a solution for us, for example a dvrOverrideMaximumWindowDuration property
All changes have now been made, larger window durations, larger chunk sizes etc. I will let you know how it works out.
In the meantime we’re investigating changing/migrating to live-record which we already happen to need for a totally different project. Perhaps we can combine these efforts and migrate to live-recording before scalability becomes an issue (mainly because of the enormous amounts of files).
In Application.xml DVR/Properties, you can add this property and change from the default value of 10800 (30 hours).
dvrMaximumRecordingLength
108000
Integer
-Lisa
I applied this to the application’s Application.xml and set it to 2600000 (a little over 30 days).
It still stopped after 30 hours!
Also, I don’t understand why it won’t cycle the material (delete the oldest content while writing the newest). Isn’t this supposed to happen instead of just stopping the recording after 30 hours?
Thank you for sharing your ideas about nDVR and details about your workflow. I will pass those on to product management as feature requests.
The limit of 30 hours is something that was not in early preview builds, but is now set by default as the maximum recording time. It is good that your planned playback duration is much smaller (e.g. 1 hour) because a recording for 30 days worth would be too large to manage and for the player to continually read. Changing the dvrChunkGroupingSeconds would allow you to customize how you want your directory structure to be maintained, but as you already mentioned, it is the total number of files that the file system has to manage that is the bigger issue. Our advice at this time is to find hardware that can support that level of file access. You could consider distributing the recording across several drives or distribute across multiple Wowza Servers to help scale. You could change the chunk size to reduce the number files generated, but it would have other consequences which may or may not be acceptable for your workflow. Because HTTP streaming has an inherent latency of 3 chunks, a larger chunk size would result in longer latency before playback would start.
The maximum limit for nDVR recording length is 30 hours, even if the WindowDuration is greater than 30 hours. The reason we do not recommend a DVR recording greater than 30 hours is twofold. The playlist will be repeated requested every couple of seconds (based on key frame size) and an extremely large playlist will be problematic. Secondly there is more risk with a longer recording and should there be a problem (network, disk full, out of memory, etc.), then the whole recording would be in jeopardy. A longer recording (say 24 hours) is okay if a smaller portion (couple of hours) of that recording is requested for playback using the playlist request API. These recommendations are based on the way HTTP and players work and how DVR can best fit into the workflow.
Another option for recording is to use StreamType live-record or for more control, use the LiveStreamRecord module.
If you can send us a zip of your corrupt nDVR store, conf and logs directories with a pointer in time to when the problems occurred, we can take a closer look. Please reference this thread and send to support@wowza.com.
You can add this property to change the default value. Again, you want to request a small portion of a long recording. Our general recommendation is not to have a recording duration longer than 30 hours. If there is any kind of problem, there is more risk when you have a very long recording. This is why we caution users against it and put in a 30 hour maximum. We suggest that you take all these factors into consideration.
Wowza nDVR relies a stable live stream with aligned audio and video. There are safeguards already built-in without additional configuration. If Wowza detects that there is an invalid chunk, it will try to fix it by recalculating if the clock was reset. Or, if it cannot fix it, the chunk will be thrown out. You will see log messages to this affect in [install-dir]/logs/wowzamedia_access.log when this occurs.
In Application.xml DVR/Properties, you can add this property and change from the default value of 10800 (30 hours).
It sounds like the configuration changes have not taken affect. Would you zip up and send [install-dir]/conf and logs to support@wowza.com so we can review the configuration settings and server logs. Please reference this thread.
