# Record stream directly to S3 with AWS Multipart Upload? **URL:** \ **Category:** Wowza Streaming Engine\ **Created:** [December 1, 2010, 10:14pm UTC](https://community.wowza.com/t/record-stream-directly-to-s3-with-aws-multipart-upload/34370 "2010-12-01T22:14:44Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![Colin\_Angevine](https://avatars.discourse-cdn.com/v4/letter/c/b5e925/32.png) [@Colin\_Angevine](https://community.wowza.com/u/Colin_Angevine)\ **Post date:** [December 1, 2010, 10:14pm UTC](https://community.wowza.com/t/record-stream-directly-to-s3-with-aws-multipart-upload/34370/1 "2010-12-01T22:14:44Z")
Hi all, I got an email from Amazon last month announcing S3 Multipart Upload: [http://aws.amazon.com/about-aws/whats-new/2010/11/10/Amazon-S3-Introducing-Multipart-Upload/](http://aws.amazon.com/about-aws/whats-new/2010/11/10/Amazon-S3-Introducing-Multipart-Upload/) Right away I started checking the Wowza forums to see if this would be rolled into the EC2-S3 configuration rather than the current implementation. Is there any chance that will be happening? Would that alleviate some of the problems that can pop up from recording to EC2 then transferring after the recording has completed? Thanks all! Ange52 ---
**Author:** ![Charlie\_Good](https://sea2.discourse-cdn.com/flex002/user_avatar/community.wowza.com/charlie_good/32/383_2.png) [@Charlie\_Good](https://community.wowza.com/u/Charlie_Good)\ **Post date:** [December 2, 2010, 4:13am UTC](https://community.wowza.com/t/record-stream-directly-to-s3-with-aws-multipart-upload/34370/2 "2010-12-02T04:13:31Z")
Recordind direct to S3 is done through the s3fs system. It would need to adopt this new addtional: [http://code.google.com/p/s3fs/](http://code.google.com/p/s3fs/) Charlie ---
**Author:** ![Charlie\_Good](https://sea2.discourse-cdn.com/flex002/user_avatar/community.wowza.com/charlie_good/32/383_2.png) [@Charlie\_Good](https://community.wowza.com/u/Charlie_Good)\ **Post date:** [December 2, 2010, 5:39am UTC](https://community.wowza.com/t/record-stream-directly-to-s3-with-aws-multipart-upload/34370/3 "2010-12-02T05:39:15Z")
We have no plans to add this type of feature. We will continue to leverage s3fs to do recording to S3. Charlie ---
**Author:** ![Richard\_Lanham](https://avatars.discourse-cdn.com/v4/letter/r/c68b51/32.png) [@Richard\_Lanham](https://community.wowza.com/u/Richard_Lanham)\ **Post date:** [January 4, 2011, 7:50pm UTC](https://community.wowza.com/t/record-stream-directly-to-s3-with-aws-multipart-upload/34370/4 "2011-01-04T19:50:35Z")
I tried using one of the current Wowza AMIs (ami-5453a53d) and it worked. I used this startup package (from [here](http://wowzamediasystems.s3.amazonaws.com/packagelist.html)): [http://wowzamediasystems.s3.amazonaws.com/com/wowza/startup/2.2/default\_2.2.zip](http://wowzamediasystems.s3.amazonaws.com/com/wowza/startup/2.2/default_2.2.zip) - Download and unzip the package - Open startup.xml in text editor, then add the init.sh script reference so it looks like this: - Add your init.sh the package (I used my AWS keys and S3 bucket of course): - Zipped the package back up - Started ami-5453a53d in ElasticFox, including the startup package by clicking Add Binary File button in Launch dialog ---
**Author:** ![Richard\_Lanham](https://avatars.discourse-cdn.com/v4/letter/r/c68b51/32.png) [@Richard\_Lanham](https://community.wowza.com/u/Richard_Lanham)\ **Post date:** [January 5, 2011, 1:38pm UTC](https://community.wowza.com/t/record-stream-directly-to-s3-with-aws-multipart-upload/34370/5 "2011-01-05T13:38:03Z")
Yes, using the configuration in that article, a new recording is moved automatically. That happens when the write process completes, not when recording stops. If you want more control, you can use the IMediaWriteActionNotifiy interface onWriteComplete handler: > **[Move recordings of live streams](https://www.wowza.com/docs/how-to-move-recordings-from-live-streams)** > > Use the MediaWriterFileMover Wowza Streaming Engine Java module to copy or move files after they've been recorded to disk. You need the [Wowza IDE](http://wowza.com/ide.html) to build application modules The source code for the automated ModuleMediaWriterFileMover is here: > [@Custom Move Module](http://community.wowza.com/t/-/35667): > > I have a custom module and I need to move a file once its recorded to an S3 mounted drive. I’m on an EC2 AMI installation of 2.1.2. When I try to move the file, I test to make sure it exists and then attempt to move the file using “renameTo” Debugging says the file DOES exist but that it “failed to move”. This happens consistently I have attempted to use the built-in filemove module but the versioning screws up the file name because it ends in numbers that are programmatically created and I can… Richard ---
**Author:** ![Richard\_Lanham](https://avatars.discourse-cdn.com/v4/letter/r/c68b51/32.png) [@Richard\_Lanham](https://community.wowza.com/u/Richard_Lanham)\ **Post date:** [January 26, 2011, 12:10am UTC](https://community.wowza.com/t/record-stream-directly-to-s3-with-aws-multipart-upload/34370/6 "2011-01-26T00:10:07Z")
Copy the command to ssh console and see it works. Double check everything. Did you add the startup.xml RunScript tag to run init.sh? It did work when I tested it. Richard ---
**Author:** ![Richard\_Lanham](https://avatars.discourse-cdn.com/v4/letter/r/c68b51/32.png) [@Richard\_Lanham](https://community.wowza.com/u/Richard_Lanham)\ **Post date:** [January 26, 2011, 3:27pm UTC](https://community.wowza.com/t/record-stream-directly-to-s3-with-aws-multipart-upload/34370/7 "2011-01-26T15:27:22Z")
I think I just used the simple bucket name: /usr/bin/s3fs cogwareplayhouston -o accessKeyId=XXXXXXXXXX -o secretAccessKey=XXXXXXXXXX -o default\_acl=public-read /mnt/s3 Your credentials provide the location, i.e., a bucket belonging to you. Richard ---
**Author:** ![Richard\_Lanham](https://avatars.discourse-cdn.com/v4/letter/r/c68b51/32.png) [@Richard\_Lanham](https://community.wowza.com/u/Richard_Lanham)\ **Post date:** [January 26, 2011, 4:12pm UTC](https://community.wowza.com/t/record-stream-directly-to-s3-with-aws-multipart-upload/34370/8 "2011-01-26T16:12:38Z")
You’re welcome, thanks for the update Richard ---
**Author:** ![Richard\_Lanham](https://avatars.discourse-cdn.com/v4/letter/r/c68b51/32.png) [@Richard\_Lanham](https://community.wowza.com/u/Richard_Lanham)\ **Post date:** [November 29, 2011, 6:57pm UTC](https://community.wowza.com/t/record-stream-directly-to-s3-with-aws-multipart-upload/34370/9 "2011-11-29T18:57:49Z")
These are associated with your Amazon AWS account. Richard ---
**Author:** ![Richard\_Lanham](https://avatars.discourse-cdn.com/v4/letter/r/c68b51/32.png) [@Richard\_Lanham](https://community.wowza.com/u/Richard_Lanham)\ **Post date:** [January 12, 2012, 6:04pm UTC](https://community.wowza.com/t/record-stream-directly-to-s3-with-aws-multipart-upload/34370/10 "2012-01-12T18:04:30Z")
Sounds like it was not mounted correctly. You have to start over. You can re-do it on the running instance. Richard ---
**Author:** ![Richard\_Lanham](https://avatars.discourse-cdn.com/v4/letter/r/c68b51/32.png) [@Richard\_Lanham](https://community.wowza.com/u/Richard_Lanham)\ **Post date:** [January 12, 2012, 6:20pm UTC](https://community.wowza.com/t/record-stream-directly-to-s3-with-aws-multipart-upload/34370/11 "2012-01-12T18:20:03Z")
There is a guide here: [https://www.wowza.com/docs/how-to-use-the-fuse-based-file-system-backed-by-amazon-s3](https://www.wowza.com/docs/how-to-use-the-fuse-based-file-system-backed-by-amazon-s3) Richard ---
**Author:** ![Richard\_Lanham](https://avatars.discourse-cdn.com/v4/letter/r/c68b51/32.png) [@Richard\_Lanham](https://community.wowza.com/u/Richard_Lanham)\ **Post date:** [January 12, 2012, 9:20pm UTC](https://community.wowza.com/t/record-stream-directly-to-s3-with-aws-multipart-upload/34370/12 "2012-01-12T21:20:57Z")
I’m not sure. I have made it work following that guide. Richard ---
**Author:** ![Ian\_Beyer](https://avatars.discourse-cdn.com/v4/letter/i/e99b99/32.png) [@Ian\_Beyer](https://community.wowza.com/u/Ian_Beyer)\ **Post date:** [April 5, 2011, 6:31am UTC](https://community.wowza.com/t/record-stream-directly-to-s3-with-aws-multipart-upload/34370/13 "2011-04-05T06:31:27Z")
If it’s mounted with s3fs, within the FTP path, yes, you should be able to see it. ---
**Author:** ![Andy\_Schnadarle](https://avatars.discourse-cdn.com/v4/letter/a/8491ac/32.png) [@Andy\_Schnadarle](https://community.wowza.com/u/Andy_Schnadarle)\ **Post date:** [November 29, 2011, 10:46am UTC](https://community.wowza.com/t/record-stream-directly-to-s3-with-aws-multipart-upload/34370/14 "2011-11-29T10:46:32Z")
Hi, Its been a while since ive used wowza server, where do i find my accessKeyId and secretAccessKey to mount the s3 bucket? ---
**Author:** ![Andy\_Schnadarle](https://avatars.discourse-cdn.com/v4/letter/a/8491ac/32.png) [@Andy\_Schnadarle](https://community.wowza.com/u/Andy_Schnadarle)\ **Post date:** [January 12, 2012, 9:52am UTC](https://community.wowza.com/t/record-stream-directly-to-s3-with-aws-multipart-upload/34370/15 "2012-01-12T09:52:43Z")
Im using the init.sh script on startup to mount the s3 bucket. My recordings are showing up on the ec2 server under: “/mnt/s3” but they are not being automatically moved to my s3 bucket. What could be the problem? ---
**Author:** ![Andy\_Schnadarle](https://avatars.discourse-cdn.com/v4/letter/a/8491ac/32.png) [@Andy\_Schnadarle](https://community.wowza.com/u/Andy_Schnadarle)\ **Post date:** [January 12, 2012, 10:09am UTC](https://community.wowza.com/t/record-stream-directly-to-s3-with-aws-multipart-upload/34370/16 "2012-01-12T10:09:07Z")
Whats the easiest way? /usr/local/bin/s3fs - when using ftp client, this directory doesn’t exist, is this required to make it work? FTP wont allow to make a new folder. Also I didn’t use - /etc/passwd-s3fs - file with the keys because its done with the init.sh file, is this ok? ---
**Author:** ![Andy\_Schnadarle](https://avatars.discourse-cdn.com/v4/letter/a/8491ac/32.png) [@Andy\_Schnadarle](https://community.wowza.com/u/Andy_Schnadarle)\ **Post date:** [January 12, 2012, 1:10pm UTC](https://community.wowza.com/t/record-stream-directly-to-s3-with-aws-multipart-upload/34370/17 "2012-01-12T13:10:39Z")
Ive been trying that guide but having trouble creating the passwd-s3fs file, and i dont want to have to do this everytime to start a ec2 instance. I am trying to set it up only using the startup package and FTP. Can you help me with that. ---
**Author:** ![Andy\_Schnadarle](https://avatars.discourse-cdn.com/v4/letter/a/8491ac/32.png) [@Andy\_Schnadarle](https://community.wowza.com/u/Andy_Schnadarle)\ **Post date:** [January 12, 2012, 3:29pm UTC](https://community.wowza.com/t/record-stream-directly-to-s3-with-aws-multipart-upload/34370/18 "2012-01-12T15:29:06Z")
The problem im having is when trying to edit “/etc/passwd-s3fs” file it says [read-only] at the bottom so it wont allow me to edit it. Im logged in as ec2-user and wont allow me to login using root. ---
**Author:** ![Andy\_Schnadarle](https://avatars.discourse-cdn.com/v4/letter/a/8491ac/32.png) [@Andy\_Schnadarle](https://community.wowza.com/u/Andy_Schnadarle)\ **Post date:** [January 12, 2012, 3:56pm UTC](https://community.wowza.com/t/record-stream-directly-to-s3-with-aws-multipart-upload/34370/19 "2012-01-12T15:56:29Z")
So everytime we start an ec2 instance and want to transfer recordings from ec2 to s3 we have to login to ssh and follow this procedure manually typing in the access code and secret access key? theres no way to automate this? ---
**Author:** ![Andy\_Schnadarle](https://avatars.discourse-cdn.com/v4/letter/a/8491ac/32.png) [@Andy\_Schnadarle](https://community.wowza.com/u/Andy_Schnadarle)\ **Post date:** [January 12, 2012, 4:31pm UTC](https://community.wowza.com/t/record-stream-directly-to-s3-with-aws-multipart-upload/34370/20 "2012-01-12T16:31:06Z")
Got the file to transfer to s3, now having trouble playing file using vods3. ```auto WARN server comment 2012-01-13 00:17:52 - - - - - 1405.995 - - - - - - - amazons3/mybucket/testing14.mp4 MediaReaderH264.open[1]: java.io.IOException: MediaCacheRandomAccessReader.open: Item not in cache: amazons3/mybucket/testing14.mp4 WARN server comment 2012-01-13 00:24:04 - - - - - 1778.166 - - - - - - - amazons3/mybucket/Extremists.flv open: java.io.IOException: MediaCacheRandomAccessReader.open: Item not in cache: amazons3/mybucket/Extremists.flv ``` [Next page](https://community.wowza.com/t/record-stream-directly-to-s3-with-aws-multipart-upload/34370.md?page=2)