# S3FS Fuse-based file system

**URL:** <https://community.wowza.com/t/s3fs-fuse-based-file-system/235>\
**Category:** Wowza Streaming Engine\
**Created:** [August 4, 2009, 6:46pm UTC](https://community.wowza.com/t/s3fs-fuse-based-file-system/235 "2009-08-04T18:46:11Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![system](https://sea2.discourse-cdn.com/flex002/user_avatar/community.wowza.com/system/32/447_2.png) [@system](https://community.wowza.com/u/system)\
**Post date:** [August 4, 2009, 6:46pm UTC](https://community.wowza.com/t/s3fs-fuse-based-file-system/235/1 "2009-08-04T18:46:11Z")

</div>

[https://www.wowza.com/docs/how-to-use-the-fuse-based-file-system-backed-by-amazon-s3](https://www.wowza.com/docs/how-to-use-the-fuse-based-file-system-backed-by-amazon-s3)

---

<div class="post-metadata">

**Author:** ![Charlie\_Good](https://sea2.discourse-cdn.com/flex002/user_avatar/community.wowza.com/charlie_good/32/383_2.png) [@Charlie\_Good](https://community.wowza.com/u/Charlie_Good)\
**Post date:** [February 16, 2010, 4:25pm UTC](https://community.wowza.com/t/s3fs-fuse-based-file-system/235/2 "2010-02-16T16:25:44Z")

</div>

[http://s3fs.googlecode.com/files/s3fs-r188-source.tar.gz](http://s3fs.googlecode.com/files/s3fs-r188-source.tar.gz)

Charlie

---

<div class="post-metadata">

**Author:** ![Tim\_Glen](https://avatars.discourse-cdn.com/v4/letter/t/c5a1d2/32.png) [@Tim\_Glen](https://community.wowza.com/u/Tim_Glen)\
**Post date:** [October 2, 2009, 3:35pm UTC](https://community.wowza.com/t/s3fs-fuse-based-file-system/235/3 "2009-10-02T15:35:59Z")

</div>

I love the s3fs concept and have been working with it to record audio without issue. I have created a script (called from startup.xml) that basically automounts an s3 bucket on startup, but the bucket doesn’t get remounted if I reboot the ec2 server. Interestingly, if I truncate the wowzamediaserverpro\_startup.log file before the reboot, no script output happens on the reboot, which leads me to believe it doesn’t get run a second time.

Here is the startup.xml file:

```auto
<Startup>
 <Commands>
    <RunScript>
      <Script>tuning/mount_s3.sh</Script>
    </RunScript>
    <Install>
      <Folder>wowza</Folder>
    </Install>
    <RunScript>
      <Script>tuning/tune.sh</Script>
    </RunScript>
  </Commands>
</Startup>

```

Here is the mount\_s3.sh file (excuse a couple of debugging echos in there):

```auto
#!/bin/sh
S3DIR="/mnt/s3"
S3BUCKET="bucket-name"
echo "Unmounting and removing the s3 recording bucket at $S3DIR if it exists"
echo "Check the contents of /mnt"
echo "`ls -l /mnt`"
if [-d $S3DIR]; then
  echo "ls -l $S3DIR"
  echo "`ls -l $S3DIR`"
  echo "/bin/fusermount -u $S3DIR"
  /bin/fusermount -u $S3DIR
else
  echo "$S3DIR doesn't exist"
fi
echo "rm -rf $S3DIR"
rm -rf $S3DIR
echo "Mounting the s3 recording bucket at $S3DIR"
echo "Creating /etc/passwd-s3fs"
echo "access_key:secret_key" > /etc/passwd-s3fs
echo "mkdir -p $S3DIR"
mkdir -p $S3DIR
echo "/usr/bin/s3fs $S3BUCKET $S3DIR"
/usr/bin/s3fs $S3BUCKET $S3DIR

```

---

<div class="post-metadata">

**Author:** ![Charlie\_Good](https://sea2.discourse-cdn.com/flex002/user_avatar/community.wowza.com/charlie_good/32/383_2.png) [@Charlie\_Good](https://community.wowza.com/u/Charlie_Good)\
**Post date:** [February 25, 2010, 9:12am UTC](https://community.wowza.com/t/s3fs-fuse-based-file-system/235/4 "2010-02-25T09:12:14Z")

</div>

You can use the variables. See the last part of the first post. It make take a bit of experimentation to get right.

Charlie

---

<div class="post-metadata">

**Author:** ![Charlie\_Good](https://sea2.discourse-cdn.com/flex002/user_avatar/community.wowza.com/charlie_good/32/383_2.png) [@Charlie\_Good](https://community.wowza.com/u/Charlie_Good)\
**Post date:** [March 9, 2010, 6:03am UTC](https://community.wowza.com/t/s3fs-fuse-based-file-system/235/5 "2010-03-09T06:03:15Z")

</div>

Make sense. We made this same change month and months ago for the vods3 stuff. I guess we will pick it up when it is integrated into the mainline. You can certainly build your own AMI now.

Charlie

---

<div class="post-metadata">

**Author:** ![Charlie\_Good](https://sea2.discourse-cdn.com/flex002/user_avatar/community.wowza.com/charlie_good/32/383_2.png) [@Charlie\_Good](https://community.wowza.com/u/Charlie_Good)\
**Post date:** [March 12, 2010, 9:19am UTC](https://community.wowza.com/t/s3fs-fuse-based-file-system/235/6 "2010-03-12T09:19:45Z")

</div>

It is a DIY project. I will give you a head start. Here is the base code:

```auto
package com.wowza.wms.module;
import java.io.*;
import java.util.*;
import com.wowza.util.*;
import com.wowza.wms.module.*;
import com.wowza.wms.application.*;
import com.wowza.wms.stream.*;
import com.wowza.wms.vhost.*;
public class ModuleMediaWriterFileMover extends ModuleBase implements IMediaWriterActionNotify
{
	String fileMoverDestinationPath = null;
	String fileMoverFileExtension = null;
	boolean fileMoverVersionFile = true;
	boolean fileMoverDeleteOriginal = false;
	
	public void onAppStart(IApplicationInstance appInstance)
	{
		appInstance.addMediaWriterListener(this);
		getLogger().info("ModuleMediaWriterFileMover.onAppStart: "+appInstance.getContextStr());
		
		WMSProperties props = appInstance.getProperties();
		
		fileMoverDestinationPath = props.getPropertyStr("fileMoverDestinationPath", fileMoverDestinationPath);
		fileMoverFileExtension = props.getPropertyStr("fileMoverFileExtension", fileMoverFileExtension);
		fileMoverVersionFile = props.getPropertyBoolean("fileMoverVersionFile", fileMoverVersionFile);
		fileMoverDeleteOriginal = props.getPropertyBoolean("fileMoverDeleteOriginal", fileMoverDeleteOriginal);
	}
	
	private String getExtension(IMediaStream stream, String ext)
	{
		if (ext == null)
		{
			String mediaReaderType = stream.getExt().toLowerCase();
			MediaReaderItem mediaReaderItem = stream.getStreams().getVHost().getMediaReaders().getMediaReaderDef(mediaReaderType);
			return mediaReaderItem.getFileExtension();
		}
		
		return ext;
	}
	public void onWriteComplete(IMediaStream stream, File file)
	{
		if (fileMoverDestinationPath != null)
		{
			String streamName = FileUtils.toValidFilename(stream.getName());
			String streamExt = getExtension(stream, fileMoverFileExtension);
			
			Map<String, String> envMap = new HashMap<String, String>();
			
			IApplicationInstance appInstance = stream.getStreams().getAppInstance();
			IVHost vhost = appInstance.getVHost();
			
			envMap.put("com.wowza.wms.context.VHost", vhost.getName());
			envMap.put("com.wowza.wms.context.VHostConfigHome", vhost.getHomePath());
			envMap.put("com.wowza.wms.context.Application", appInstance.getApplication().getName());
			envMap.put("com.wowza.wms.context.ApplicationInstance", appInstance.getName());
			envMap.put("com.wowza.wms.context.StreamName", streamName);
			
			String destinationPath = SystemUtils.expandEnvironmentVariables(fileMoverDestinationPath, envMap);
			
			if (fileMoverDestinationPath.indexOf("com.wowza.wms.context.StreamName") < 0)
				destinationPath += "/"+streamName+"."+streamExt;
			
			File dstFile = new File(destinationPath);
			
			getLogger().info("ModuleMediaWriterFileMover.onWriteComplete: from: "+file+" to:"+dstFile);
	
			try
			{
				dstFile.getParentFile().mkdirs();
			}
			catch(Exception e)
			{
				getLogger().warn("ModuleMediaWriterFileMover.onWriteComplete[mkdir]: ["+dstFile+"]: "+e.toString());
			}
			
			try
			{
				if (dstFile.exists())
				{
					if (fileMoverVersionFile)
						FileUtils.versionFile(dstFile);
					else
						dstFile.delete();
				}
			}
			catch(Exception e)
			{
				getLogger().warn("ModuleMediaWriterFileMover.onWriteComplete[version]: ["+dstFile+"]: "+e.toString());
			}
			
			try
			{
				FileUtils.copyFile2(file, dstFile);
			}
			catch(Exception e)
			{
				getLogger().warn("ModuleMediaWriterFileMover.onWriteComplete[version]: ["+dstFile+"]: "+e.toString());
			}
	
			try
			{
				if (fileMoverDeleteOriginal && dstFile.exists())
				{
					file.delete();
				}
			}
			catch(Exception e)
			{
				getLogger().warn("ModuleMediaWriterFileMover.onWriteComplete[delOrig]: ["+dstFile+"]: "+e.toString());
			}
		}
	}
	
	public void onFLVAddMetadata(IMediaStream stream, Map<String, Object> extraMetadata)
	{
	}
}

```

Charlie

---

<div class="post-metadata">

**Author:** ![Charlie\_Good](https://sea2.discourse-cdn.com/flex002/user_avatar/community.wowza.com/charlie_good/32/383_2.png) [@Charlie\_Good](https://community.wowza.com/u/Charlie_Good)\
**Post date:** [September 8, 2010, 4:03am UTC](https://community.wowza.com/t/s3fs-fuse-based-file-system/235/7 "2010-09-08T04:03:08Z")

</div>

We do not have a method for sharing shard object between servers.

Charlie

---

<div class="post-metadata">

**Author:** ![Charlie\_Good](https://sea2.discourse-cdn.com/flex002/user_avatar/community.wowza.com/charlie_good/32/383_2.png) [@Charlie\_Good](https://community.wowza.com/u/Charlie_Good)\
**Post date:** [September 8, 2010, 4:18am UTC](https://community.wowza.com/t/s3fs-fuse-based-file-system/235/8 "2010-09-08T04:18:30Z")

</div>

I do not know if this will work. I have never tried it.

Charlie

---

<div class="post-metadata">

**Author:** ![Richard\_Lanham](https://avatars.discourse-cdn.com/v4/letter/r/c68b51/32.png) [@Richard\_Lanham](https://community.wowza.com/u/Richard_Lanham)\
**Post date:** [February 11, 2010, 9:17pm UTC](https://community.wowza.com/t/s3fs-fuse-based-file-system/235/9 "2010-02-11T21:17:05Z")

</div>

Yes, EBS does not have this problem, it supposed to have better throughput then the instance local storage. That’s what Amazon says on their EBS site. And it is an excellent way to insulate yourself from instance failure.

Richard

---

<div class="post-metadata">

**Author:** ![Richard\_Lanham](https://avatars.discourse-cdn.com/v4/letter/r/c68b51/32.png) [@Richard\_Lanham](https://community.wowza.com/u/Richard_Lanham)\
**Post date:** [February 16, 2010, 10:33am UTC](https://community.wowza.com/t/s3fs-fuse-based-file-system/235/10 "2010-02-16T10:33:06Z")

</div>

Using ls command is the test, so if that’s not working something is not right.

Which AMI are you using? I used ami-ff8b6796 yesterday to try out the new ModuleMediaWriterFileMover, and it worked great.

Check your AWS keys.

Richarrd

---

<div class="post-metadata">

**Author:** ![Richard\_Lanham](https://avatars.discourse-cdn.com/v4/letter/r/c68b51/32.png) [@Richard\_Lanham](https://community.wowza.com/u/Richard_Lanham)\
**Post date:** [February 16, 2010, 1:13pm UTC](https://community.wowza.com/t/s3fs-fuse-based-file-system/235/11 "2010-02-16T13:13:49Z")

</div>

Try it without the sub-bucket, and without the extra args. I assume you are replacing with your access and secret key.

Richard

---

<div class="post-metadata">

**Author:** ![Richard\_Lanham](https://avatars.discourse-cdn.com/v4/letter/r/c68b51/32.png) [@Richard\_Lanham](https://community.wowza.com/u/Richard_Lanham)\
**Post date:** [February 16, 2010, 1:48pm UTC](https://community.wowza.com/t/s3fs-fuse-based-file-system/235/12 "2010-02-16T13:48:51Z")

</div>

It will not work to create a “sub-folder” with mkdir command from linux. It has to be a proper sub-bucket created in your S3 interface.

Richard

---

<div class="post-metadata">

**Author:** ![Richard\_Lanham](https://avatars.discourse-cdn.com/v4/letter/r/c68b51/32.png) [@Richard\_Lanham](https://community.wowza.com/u/Richard_Lanham)\
**Post date:** [February 16, 2010, 6:28pm UTC](https://community.wowza.com/t/s3fs-fuse-based-file-system/235/13 "2010-02-16T18:28:10Z")

</div>

I tried it out, seems that sub-buckets or subfolders created with s3 plugin are useless. You can create different mount points. You’re not going to be able to do subfolders.

Richard

---

<div class="post-metadata">

**Author:** ![Richard\_Lanham](https://avatars.discourse-cdn.com/v4/letter/r/c68b51/32.png) [@Richard\_Lanham](https://community.wowza.com/u/Richard_Lanham)\
**Post date:** [February 16, 2010, 11:36pm UTC](https://community.wowza.com/t/s3fs-fuse-based-file-system/235/14 "2010-02-16T23:36:01Z")

</div>

Thanks for the suggestion. I don’t think any other 3rd party tools are likely to be included anytime soon.

Richard

---

<div class="post-metadata">

**Author:** ![Richard\_Lanham](https://avatars.discourse-cdn.com/v4/letter/r/c68b51/32.png) [@Richard\_Lanham](https://community.wowza.com/u/Richard_Lanham)\
**Post date:** [March 9, 2010, 1:40pm UTC](https://community.wowza.com/t/s3fs-fuse-based-file-system/235/15 "2010-03-09T13:40:28Z")

</div>

I don’t know why, but I think this solves the problem, it won’t affect the application significantly.

---

<div class="post-metadata">

**Author:** ![Richard\_Lanham](https://avatars.discourse-cdn.com/v4/letter/r/c68b51/32.png) [@Richard\_Lanham](https://community.wowza.com/u/Richard_Lanham)\
**Post date:** [June 10, 2010, 6:25pm UTC](https://community.wowza.com/t/s3fs-fuse-based-file-system/235/16 "2010-06-10T18:25:15Z")

</div>

Take a look at this post:

[https://www.wowza.com/forums/showthread.php?p=39370#5](https://www.wowza.com/forums/showthread.php?p=39370#5)

Richard

---

<div class="post-metadata">

**Author:** ![Richard\_Lanham](https://avatars.discourse-cdn.com/v4/letter/r/c68b51/32.png) [@Richard\_Lanham](https://community.wowza.com/u/Richard_Lanham)\
**Post date:** [June 10, 2010, 7:15pm UTC](https://community.wowza.com/t/s3fs-fuse-based-file-system/235/17 "2010-06-10T19:15:15Z")

</div>

About ftp created directories in s3. I don’t that’s a good idea. It’s like using mkdir on s3fs mounted bucket. It might look like it creates something, but it’s not usable. You have to use an S3 tool or EC2 tools to create buckets and “sub-buckets”

Richard

---

<div class="post-metadata">

**Author:** ![Richard\_Lanham](https://avatars.discourse-cdn.com/v4/letter/r/c68b51/32.png) [@Richard\_Lanham](https://community.wowza.com/u/Richard_Lanham)\
**Post date:** [June 10, 2010, 9:17pm UTC](https://community.wowza.com/t/s3fs-fuse-based-file-system/235/18 "2010-06-10T21:17:38Z")

</div>

We only recommend s3fs for this one use case, copy file to s3 after recording:

[http://community.wowza.com/t/-/235](http://community.wowza.com/t/-/235)

Richard

---

<div class="post-metadata">

**Author:** ![Richard\_Lanham](https://avatars.discourse-cdn.com/v4/letter/r/c68b51/32.png) [@Richard\_Lanham](https://community.wowza.com/u/Richard_Lanham)\
**Post date:** [August 5, 2010, 12:41am UTC](https://community.wowza.com/t/s3fs-fuse-based-file-system/235/19 "2010-08-05T00:41:24Z")

</div>

We will look into it. I can’t provide a time frame, but it’s on the todo list now.

Richard

---

<div class="post-metadata">

**Author:** ![Richard\_Lanham](https://avatars.discourse-cdn.com/v4/letter/r/c68b51/32.png) [@Richard\_Lanham](https://community.wowza.com/u/Richard_Lanham)\
**Post date:** [August 10, 2010, 12:14pm UTC](https://community.wowza.com/t/s3fs-fuse-based-file-system/235/20 "2010-08-10T12:14:30Z")

</div>

I suggest building ffmpeg and other such things on a EBS volume, which you can attach to a new instance and snapshot as necessary.

Richard

[Next page](https://community.wowza.com/t/s3fs-fuse-based-file-system/235.md?page=2)