Do MP4 files for playback via VOD need to be on the Wowza box itself (in the “content” folder or a custom folder) or can these be hosted remotely?
Conversely, do I have to save the MP4 to local disk when saving a live stream, or can they be saved to a remote location…?
The can be hosted remotely; e.g. in a NAS, S3 bucket, Google Cloud Storage, Azure Storage, etc. as long as you can mount the remote storage as a volume on your server or it’s supported by MediaCache (see below). If you want to pull assets over FTP or other file transfer technology, you’ll have to do that via a custom plug-in.
The challenge with remote storage is always the bandwidth. With local storage usually one assumes that the IOPS are sufficient for servicing the concurrent VOD streams, although it can be a concern in environments under heavy load.
But with remote storage you will be dependent on the available bandwidth and the connection speed, and particularly when the load increases you’ll have to keep an eye on the performance and maybe implement caching or consider horizontal scaling. (“load” being: the number of assets that is concurrently being read from disk and their respective bit rates)
Remember also that adding more VOD servers won’t always help if the pipe to the remote storage is shared.
You can store the MP4 on remote storage, e.g. AWS S3 Bucket or Google Cloud Storage and enable MediaCache, which is a built-in feature (see article below for all supported storage types). That will give some relief from caching; but it only works if the load comes from the same assets being viewed over and over. Caching logically won’t help if 1000 users watch 1000 different assets.
See this article for more info on MediaCache (the term “Scale” in the title is debatable though): https://www.wowza.com/docs/how-to-scale-video-on-demand-streaming-with-media-cache