Hi everyone. I have an issue with the movement of recorded files to S3. Sometimes, actually quite rarely, I catch the following exceptions:
ERROR server comment 2016-03-18 05:40:47 - - - - - 117788.092 - - - - - - - - FileUtils.copyFile2 : java.io.FileNotFoundException: /mnt/s3/camera_4_6fa289c4-cfba-4581-ab5b-82b15dcbe48c.stream_2016_03_18_053945375.mp4 (No such file or directory)|at java.io.FileOutputStream.open0(Native Method)|at java.io.FileOutputStream.open(FileOutputStream.java:270)|at java.io.FileOutputStream.<init>(FileOutputStream.java:213)|at java.io.FileOutputStream.<init>(FileOutputStream.java:162)|at com.wowza.util.FileUtils.copyFile2(FileUtils.java:153)|
sometimes it’s a single exception as in the example above, and sometimes it appears together with other exceptions as in the example below
ERROR server comment 2016-03-24 12:18:34 - - - - - 83973.669 - - - - - - - - Cannot execute query: java.sql.SQLException: Could not retrieve transation read-only status server|at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1084)|at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:987)|at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:973)|at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:918)|at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:949)|
ERROR server comment 2016-03-24 12:18:34 - - - - - 83973.672 - - - - - - - - [live/_definst_]LiveStreamRecorderMP4: wowza35_6844f9d6-f798-45b1-b42b-276ce02ee672.stream: /usr/local/WowzaStreamingEngine-4.3.0/content/wowza35_6844f9d6-f798-45b1-b42b-276ce02ee672.stream_2016_03_24_121733829.mp4: java.lang.RuntimeException: Cannot execute query|at com.l1inc.wowzacustomize.RecorderWriteListener.onSegmentEnd(RecorderWriteListener.java:73)|at com.wowza.wms.livestreamrecord.manager.StreamRecorder.notifySegmentEnd(StreamRecorder.java:754)|at com.wowza.wms.livestreamrecord.manager.StreamRecorder$LiveStreamRecordActionListener.onSegmentEnd(StreamRecorder.java:58)|at com.wowza.wms.livestreamrecord.model.LiveStreamRecorderBase.notifySegmentEnd(LiveStreamRecorderBase.java:568)|at com.wowza.wms.livestreamrecord.model.LiveStreamRecorderMP4.b(LiveStreamRecorderMP4.java:138)|
ERROR server comment 2016-03-24 12:18:34 - - - - - 83973.678 - - - - - - - - Cannot execute query: java.sql.SQLException: Could not retrieve transation read-only status server|at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1084)|at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:987)|at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:973)|at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:918)|at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:949)|
ERROR server comment 2016-03-24 12:18:36 - - - - - 83975.554 - - - - - - - - FileUtils.copyFile2 : java.io.FileNotFoundException: /usr/local/WowzaStreamingEngine-4.3.0/content/wowza35_6844f9d6-f798-45b1-b42b-276ce02ee672.stream_2016_03_24_121733829.mp4 (No such file or directory)|at java.io.FileInputStream.open0(Native Method)|at java.io.FileInputStream.open(FileInputStream.java:195)|at java.io.FileInputStream.<init>(FileInputStream.java:138)|at com.wowza.util.FileUtils.copyFile2(FileUtils.java:152)|at com.wowza.wms.module.ModuleMediaWriterFileMover$CopyWorker.run(ModuleMediaWriterFileMover.java:84)|
As you can see I use WowzaStreamingEngine-4.3.0.
I use several additional modules:
-
com.wowza.wms.module.ModuleMediaWriterFileMover to move files to the mounted S3 bucket.
-
custom FileNameDelegate to rename recorded files
-
custom RecorderWriteListener to capture information about each recorded segment and to save this information in a database.
Here are examples of these modules:
public class FileNameDelegate extends ModuleBase implements IStreamRecorderFileVersionDelegate {
static DateFormat dateFormat = new SimpleDateFormat("yyyy_MM_dd_HHmmssSSS");
public String getFilename(IStreamRecorder recorder) {
String name;
Date date = new Date();
try {
File file = new File(recorder.getBaseFilePath());
String oldBasePath = file.getParent();
String oldName = file.getName();
String oldExt = "";
int oldExtIndex = oldName.lastIndexOf(".");
if (oldExtIndex >= 0) {
oldExt = oldName.substring(oldExtIndex);
oldName = oldName.substring(0, oldExtIndex);
}
if ((file = new File(name = String.valueOf(oldBasePath) + "/" + oldName + "_" + dateFormat.format(date) + oldExt)).exists()) {
file.delete();
}
}
catch (Exception e) {
getLogger().error("MyDelegate.getFilename: " + e.toString());
name = "temp" + dateFormat.format(date) + ".tmp";
throw new RuntimeException(e);
}
return name;
}
}
public class RecorderWriteListener extends ModuleBase implements IStreamRecorderActionNotify {
private static final String DB_URL = "dnaDBurl";
private static final String DB_USER = "dnaDBuser";
private static final String DB_PWD = "dnaDBpwd";
private Long startRecordTime = null;
private Long endRecordTime = null;
private static ComboPooledDataSource cpds;
private static final String INSERT_QUERY = "INSERT INTO dnamain.CameraRecordFile (filename, id_camera, starttime, endtime, duration, filesize) " +
"SELECT ?, c.id, ?, ?, ?, ? FROM dnamain.Camera c WHERE c.streamname = ?";
public void onAppStart(IApplicationInstance appInstance) {
initDBConnection(appInstance);
}
public void onAppStop(IApplicationInstance appInstance) {
destroyDBConnection();
}
@Override
public void onSegmentEnd(IStreamRecorder recorder) {
if (startRecordTime == null)
startRecordTime = recorder.getStartTime().getMillis();
String fileName = recorder.getCurrentFile().replaceAll("/usr/local/WowzaStreamingEngine-4.3.0/content/", "");
String streamName = recorder.getStreamName().replaceAll(".stream", "");
Timestamp startTime = new Timestamp(startRecordTime);
Long duration = recorder.getCurrentDuration();
endRecordTime = startRecordTime + duration;
Timestamp endTime = new Timestamp(endRecordTime);
Long fileSize = recorder.getCurrentSize();
startRecordTime = endRecordTime;
try (Connection connection = cpds.getConnection()) {
try (PreparedStatement insertStatement = connection.prepareStatement(INSERT_QUERY)) {
try {
insertStatement.setString(1, fileName);
insertStatement.setTimestamp(2, startTime);
insertStatement.setTimestamp(3, endTime);
insertStatement.setLong(4, duration);
insertStatement.setLong(5, fileSize);
insertStatement.setString(6, streamName);
} catch (SQLException e1) {
getLogger().error("Cannot set parameter", e1);
throw new RuntimeException("Cannot set parameter", e1);
}
try {
insertStatement.executeUpdate();
} catch (SQLException e) {
getLogger().error("Cannot execute query", e);
throw new RuntimeException("Cannot execute query", e);
}
} catch (SQLException e) {
getLogger().error("cannot prepare statement", e);
throw new RuntimeException(e);
}
} catch (SQLException e) {
getLogger().error("cannot open connection", e);
throw new RuntimeException(e);
}
}
@Override
public void onCreateRecorder(IStreamRecorder recorder) {
}
@Override
public void onSegmentStart(IStreamRecorder recorder) {
}
@Override
public void onSplitRecorder(IStreamRecorder recorder) {
}
@Override
public void onStartRecorder(IStreamRecorder recorder) {
}
@Override
public void onStopRecorder(IStreamRecorder recorder) {
}
@Override
public void onSwitchRecorder(IStreamRecorder recorder, IMediaStream newStream) {
}
private void initDBConnection(IApplicationInstance appInstance) {
WMSProperties wmfp = appInstance.getProperties();
String dbURL = wmfp.getPropertyStr(DB_URL);
String dbUser = wmfp.getPropertyStr(DB_USER);
String dbPwd = wmfp.getPropertyStr(DB_PWD);
if (cpds == null) {
cpds = new ComboPooledDataSource();
}
try {
cpds.setDriverClass("com.mysql.jdbc.Driver");
} catch (PropertyVetoException e) {
getLogger().error("Cannot load JDBC driver", e);
throw new RuntimeException(e);
}
cpds.setJdbcUrl(dbURL);
cpds.setUser(dbUser);
cpds.setPassword(dbPwd);
cpds.setDebugUnreturnedConnectionStackTraces(true);
cpds.setUnreturnedConnectionTimeout(10);
}
private void destroyDBConnection() {
if (cpds != null) {
cpds.close();
cpds = null;
}
}
}
And live/Application.xml example
<?xml version="1.0" encoding="UTF-8"?>
<Root version="1">
<Application>
<Name>live</Name>
<AppType>Live</AppType>
<Description></Description>
<!-- Uncomment to set application level timeout values
<ApplicationTimeout>60000</ApplicationTimeout>
<PingTimeout>12000</PingTimeout>
<ValidationFrequency>8000</ValidationFrequency>
<MaximumPendingWriteBytes>0</MaximumPendingWriteBytes>
<MaximumSetBufferTime>60000</MaximumSetBufferTime>
<MaximumStorageDirDepth>25</MaximumStorageDirDepth>
-->
<Connections>
<AutoAccept>true</AutoAccept>
<AllowDomains></AllowDomains>
</Connections>
<!--
StorageDir path variables
${com.wowza.wms.AppHome} - Application home directory
${com.wowza.wms.ConfigHome} - Configuration home directory
${com.wowza.wms.context.VHost} - Virtual host name
${com.wowza.wms.context.VHostConfigHome} - Virtual host config directory
${com.wowza.wms.context.Application} - Application name
${com.wowza.wms.context.ApplicationInstance} - Application instance name
-->
<Streams>
<StreamType>live</StreamType>
<StorageDir>${com.wowza.wms.context.VHostConfigHome}/content</StorageDir>
<KeyDir>${com.wowza.wms.context.VHostConfigHome}/keys</KeyDir>
<!-- LiveStreamPacketizers (separate with commas): cupertinostreamingpacketizer, smoothstreamingpacketizer, sanjosestreamingpacketizer, mpegdashstreamingpacketizer, cupertinostreamingrepeater, smoothstreamingrepeater, sanjosestreamingrepeater, mpegdashstreamingrepeater, dvrstreamingpacketizer, dvrstreamingrepeater -->
<LiveStreamPacketizers>cupertinostreamingpacketizer, mpegdashstreamingpacketizer</LiveStreamPacketizers>
<!-- Properties defined here will override any properties defined in conf/Streams.xml for any streams types loaded by this application -->
<Properties>
</Properties>
</Streams>
<Transcoder>
<!-- To turn on transcoder set to: transcoder -->
<LiveStreamTranscoder></LiveStreamTranscoder>
<!-- [templatename].xml or ${SourceStreamName}.xml -->
<Templates>${SourceStreamName}.xml,transrate.xml</Templates>
<ProfileDir>${com.wowza.wms.context.VHostConfigHome}/transcoder/profiles</ProfileDir>
<TemplateDir>${com.wowza.wms.context.VHostConfigHome}/transcoder/templates</TemplateDir>
<Properties>
</Properties>
</Transcoder>
<DVR>
<!-- As a single server or as an origin, use dvrstreamingpacketizer in LiveStreamPacketizers above -->
<!-- Or, in an origin-edge configuration, edges use dvrstreamingrepeater in LiveStreamPacketizers above -->
<!-- As an origin, also add dvrchunkstreaming to HTTPStreamers below -->
<!-- If this is a dvrstreamingrepeater, define Application/Repeater/OriginURL to point back to the origin -->
<!-- To turn on DVR recording set Recorders to dvrrecorder. This works with dvrstreamingpacketizer -->
<Recorders></Recorders>
<!-- As a single server or as an origin, set the Store to dvrfilestorage-->
<!-- edges should have this empty -->
<Store></Store>
<!-- Window Duration is length of live DVR window in seconds. 0 means the window is never trimmed. -->
<WindowDuration>0</WindowDuration>
<!-- Storage Directory is top level location where dvr is stored. e.g. c:/temp/dvr -->
<StorageDir>${com.wowza.wms.context.VHostConfigHome}/dvr</StorageDir>
<!-- valid ArchiveStrategy values are append, version, delete -->
<ArchiveStrategy>append</ArchiveStrategy>
<!-- Properties for DVR -->
<Properties>
</Properties>
</DVR>
<TimedText>
<!-- VOD caption providers (separate with commas): vodcaptionprovidermp4_3gpp, vodcaptionproviderttml, vodcaptionproviderwebvtt, vodcaptionprovidersrt, vodcaptionproviderscc -->
<VODTimedTextProviders></VODTimedTextProviders>
<!-- Properties for TimedText -->
<Properties>
</Properties>
</TimedText>
<!-- HTTPStreamers (separate with commas): cupertinostreaming, smoothstreaming, sanjosestreaming, mpegdashstreaming, dvrchunkstreaming -->
<HTTPStreamers>cupertinostreaming, mpegdashstreaming</HTTPStreamers>
<MediaCache>
<MediaCacheSourceList></MediaCacheSourceList>
</MediaCache>
<SharedObjects>
<StorageDir>${com.wowza.wms.context.VHostConfigHome}/applications/${com.wowza.wms.context.Application}/sharedobjects/${com.wowza.wms.context.ApplicationInstance}</StorageDir>
</SharedObjects>
<Client>
<IdleFrequency>-1</IdleFrequency>
<Access>
<StreamReadAccess></StreamReadAccess>
<StreamWriteAccess>*</StreamWriteAccess>
<StreamAudioSampleAccess></StreamAudioSampleAccess>
<StreamVideoSampleAccess></StreamVideoSampleAccess>
<SharedObjectReadAccess>*</SharedObjectReadAccess>
<SharedObjectWriteAccess>*</SharedObjectWriteAccess>
</Access>
</Client>
<RTP>
<!-- RTP/Authentication/[type]Methods defined in Authentication.xml. Default setup includes; none, basic, digest -->
<Authentication>
<PublishMethod>digest</PublishMethod>
<PlayMethod>block</PlayMethod>
</Authentication>
<!-- RTP/AVSyncMethod. Valid values are: senderreport, systemclock, rtptimecode -->
<AVSyncMethod>senderreport</AVSyncMethod>
<MaxRTCPWaitTime>12000</MaxRTCPWaitTime>
<IdleFrequency>75</IdleFrequency>
<RTSPSessionTimeout>90000</RTSPSessionTimeout>
<RTSPMaximumPendingWriteBytes>0</RTSPMaximumPendingWriteBytes>
<RTSPBindIpAddress></RTSPBindIpAddress>
<RTSPConnectionIpAddress>0.0.0.0</RTSPConnectionIpAddress>
<RTSPOriginIpAddress>127.0.0.1</RTSPOriginIpAddress>
<IncomingDatagramPortRanges>*</IncomingDatagramPortRanges>
<!-- Properties defined here will override any properties defined in conf/RTP.xml for any depacketizers loaded by this application -->
<Properties>
</Properties>
</RTP>
<MediaCaster>
<RTP>
<RTSP>
<!-- udp, interleave -->
<RTPTransportMode>interleave</RTPTransportMode>
</RTSP>
</RTP>
<StreamValidator>
<Enable>true</Enable>
<ResetNameGroups>true</ResetNameGroups>
<StreamStartTimeout>20000</StreamStartTimeout>
<StreamTimeout>20000</StreamTimeout>
<VideoStartTimeout>0</VideoStartTimeout>
<VideoTimeout>0</VideoTimeout>
<AudioStartTimeout>0</AudioStartTimeout>
<AudioTimeout>0</AudioTimeout>
<VideoTCToleranceEnable>false</VideoTCToleranceEnable>
<VideoTCPosTolerance>3000</VideoTCPosTolerance>
<VideoTCNegTolerance>-500</VideoTCNegTolerance>
<AudioTCToleranceEnable>false</AudioTCToleranceEnable>
<AudioTCPosTolerance>3000</AudioTCPosTolerance>
<AudioTCNegTolerance>-500</AudioTCNegTolerance>
<DataTCToleranceEnable>false</DataTCToleranceEnable>
<DataTCPosTolerance>3000</DataTCPosTolerance>
<DataTCNegTolerance>-500</DataTCNegTolerance>
<AVSyncToleranceEnable>false</AVSyncToleranceEnable>
<AVSyncTolerance>1500</AVSyncTolerance>
<DebugLog>false</DebugLog>
</StreamValidator>
<!-- Properties defined here will override any properties defined in conf/MediaCasters.xml for any MediaCasters loaded by this applications -->
<Properties>
</Properties>
</MediaCaster>
<MediaReader>
<!-- Properties defined here will override any properties defined in conf/MediaReaders.xml for any MediaReaders loaded by this applications -->
<Properties>
</Properties>
</MediaReader>
<MediaWriter>
<!-- Properties defined here will override any properties defined in conf/MediaWriter.xml for any MediaWriter loaded by this applications -->
<Properties>
</Properties>
</MediaWriter>
<LiveStreamPacketizer>
<!-- Properties defined here will override any properties defined in conf/LiveStreamPacketizers.xml for any LiveStreamPacketizers loaded by this applications -->
<Properties>
</Properties>
</LiveStreamPacketizer>
<HTTPStreamer>
<!-- Properties defined here will override any properties defined in conf/HTTPStreamers.xml for any HTTPStreamer loaded by this applications -->
<Properties>
</Properties>
</HTTPStreamer>
<Manager>
<!-- Properties defined are used by the Manager -->
<Properties>
</Properties>
</Manager>
<Repeater>
<OriginURL></OriginURL>
<QueryString><![CDATA[]]></QueryString>
</Repeater>
<StreamRecorder>
<Properties>
<Property>
<Name>streamRecorderFileVersionDelegate</Name>
<Value>com.l1inc.wowzacustomize.FileNameDelegate</Value>
<Type>String</Type>
</Property>
<Property>
<Name>streamRecorderNotifyListener</Name>
<Value>com.l1inc.wowzacustomize.RecorderWriteListener</Value>
<Type>String</Type>
</Property>
<Property>
<Name>streamRecorderDebugEnable</Name>
<Value>true</Value>
<Type>Boolean</Type>
</Property>
<Property>
<Name>streamRecorderStartOnKeyFrame</Name>
<Value>true</Value>
<Type>Boolean</Type>
</Property>
<Property>
<Name>streamRecorderMoveFirstVideoFrameToZero</Name>
<Value>true</Value>
<Type>Boolean</Type>
</Property>
<Property>
<Name>streamRecorderSplitOnTcDiscontinuity</Name>
<Value>true</Value>
<Type>Boolean</Type>
</Property>
<Property>
<Name>streamRecorderVersioningOption</Name>
<Value>version</Value>
<Type>String</Type>
</Property>
</Properties>
</StreamRecorder>
<Modules>
<Module>
<Name>base</Name>
<Description>Base</Description>
<Class>com.wowza.wms.module.ModuleCore</Class>
</Module>
<Module>
<Name>logging</Name>
<Description>Client Logging</Description>
<Class>com.wowza.wms.module.ModuleClientLogging</Class>
</Module>
<Module>
<Name>flvplayback</Name>
<Description>FLVPlayback</Description>
<Class>com.wowza.wms.module.ModuleFLVPlayback</Class>
</Module>
<Module>
<Name>ModuleCoreSecurity</Name>
<Description>Core Security Module for Applications</Description>
<Class>com.wowza.wms.security.ModuleCoreSecurity</Class>
</Module>
<Module>
<Name>RecorderWriteListener</Name>
<Description>RecorderWriteListener</Description>
<Class>com.l1inc.wowzacustomize.RecorderWriteListener</Class>
</Module>
<Module>
<Name>FileNameDelegate</Name>
<Description>FileNameDelegate</Description>
<Class>com.l1inc.wowzacustomize.FileNameDelegate</Class>
</Module>
<Module>
<Name>AutoRecordStreams</Name>
<Description>AutoRecordStreams</Description>
<Class>com.l1inc.wowzacustomize.AutoRecordStreams</Class>
</Module>
<Module>
<Name>ModuleMediaWriterFileMover</Name>
<Description>ModuleMediaWriterFileMover</Description>
<Class>com.wowza.wms.module.ModuleMediaWriterFileMover</Class>
</Module>
</Modules>
<!-- Properties defined here will be added to the IApplication.getProperties() and IApplicationInstance.getProperties() collections -->
<Properties>
<Property>
<Name>securityPublishRequirePassword</Name>
<Value>true</Value>
<Type>Boolean</Type>
</Property>
<Property>
<Name>fileMoverDestinationPath</Name>
<Value>/mnt/s3</Value>
</Property>
<Property>
<Name>fileMoverDeleteOriginal</Name>
<Value>true</Value>
<Type>Boolean</Type>
</Property>
<Property>
<Name>fileMoverVersionFile</Name>
<Value>true</Value>
<Type>Boolean</Type>
</Property>
<Property>
<Name>dnaDBurl</Name>
<Value>jdbc:mysql://*********:3306</Value>
<Type>String</Type>
</Property>
<Property>
<Name>dnaDBuser</Name>
<Value>*******</Value>
<Type>String</Type>
</Property>
<Property>
<Name>dnaDBpwd</Name>
<Value>*******</Value>
<Type>String</Type>
</Property>
</Properties>
</Application>
</Root>
By the way I found recorded file from second exception example in the S3 bucket and I didn’t find it in the local folder. So it seems that the file was moved, but it had versioned name like “originalName_0.mp4” as if it was recorded twice. Although it should not be so. Also nothing had been saved in the database.
And when “fileMoverVersionFile” property was set to “false” I could find one file in the S3 and another in the local folder. Both files had the same name, but in the most cases one of them had the size 0 bytes. And there was saved information about this file in the database. The exceptions were the same as now.
I have no idea why it’s happening. Could anybody explain me what I’m doing wrong?
Thanks!