Suggestion/Request: A restructure of Wowza'a directory Layout (vhosts spersific)

Lee_Wickham · December 31, 2014, 9:43pm

Hey every one,

This suggestion is primarily based for those of us who use Wowza in a multiple vHost configuration, though I believe it would benefit the application as a whole in the long run.

Following the erlier suggestion I made regarding Reloading Wowza’s configuration files on the fly with out restarting, I have additionally found that Wowza’s directoy layout to be somewhat confusing where vHost’s are concerned.

Updates:

Updated the custom logging settings - thease settings now work as desired allowing for dynamic vHost naming conventions (no need to further edit/update the file if you decide to move the default vHost config folder)
Updated/added - Description of a current bug/flaw when enablingCustom Password File location, that causes the publish.password file to be written to globaly across all live vHosts.

Note: This is based on my first initial impressions of the system and as sutch i am writeing this form the prespective of a New/beginner Administrator attempting to setup a new vHost after reading through a few documentation articles, I have since done a lot more indepth readin and trial and error figuring out what goes where and why, however I still beleive the point i am making below is vlaid and would generally assist all users old and new when using wowza in the way documented.

The Problem:
Duplicating the setup of the “defaultVHost” in the process of creating a new vHost instance.

Having all of the files and folders vHost & application core together in the same directory layout and not being able to, with out some trial and error, easely see what is vHost related and what is Core Application related.

The Reason this is a problem:
Very Slow vHosts Instance creation for new/beginner Administrators, This is due to the following:
The new/beginner Administrator by just looking at the current application layout is unable to easely identify Core Application layout from vHost Layout (files and folders) This can lead to:
The administration easily missing critical files or folders thus causing runtime errors that they then need to go and lookup in their logs and backtrace to find whats missing.
The administration easily copying files or folders that are redundant, not needed or part of the Core Application into their new vHost thus bloating it out and/or wasting space.

Suggested Solution:
Note: This is actually first step I now perform when setting up my testing/development and eventualy my production environment, it just keeps every thing clean tidy and compartmentalised.
Separate the Core Application layout from the vHost layout, this can be done very easily by simply moving the default and subsequent vHost’s into their own sub-folder. (example shown below).

Setup a template for vHost creation so that when creating a new vHost you can simply just referance the templace and have all files and folders generated as needed, in the correct location.

Suggested Improved Directory Structure:
Note: Custom Password File location - There is currently a bug that occurs when editing the publishers via the webmanager it will globally write to all publish.password files across all currently live vHosts.

if this could be changed to only write to the file associated with the vHost you are currently administrating this would solve a lot of problems here.

Aadditionally I would love to see this option be able to be get vHost wide as well as of appliction wide, gives us end users a lot more flexability and customizability.

        Wowza Core Directory Layout:
            backup/
            bin/
            conf/ - All core related config and VHost.xml file all other related vHost files are moved to their respective new locations.
                VHosts.xml - global vHost configuration file, amended for new layout, see example's below.
                log4j.properties - updated to allow for each vHost to have its own separate access, error & statistics logging.
              
            documentation/
            examples/
            legal/
            lib/
            lib-native/
            logs/ - for logging non vHost related issues.
            manager/
            stats/  - for logging non vHost related statistics.
            updates/
            vhosts/ - New location for all vHost's - Default vhost automatically created here, all subsiquent vHosts have their default location set to this location.
      
        Wowza Vhost Diretory Layout:
            vhosts/
                vhost001/ - _default_ vHost
                    applications/
                    conf/
                        VHost.xml - updated to include change in folder layout.
                        vod/
                        live/
                        applicationname/
                        ...
                    content/
                        vod/
                        live/
                        applicationname/
                        ...
                    dvr/
                        dvr/
                            _definst_
                    keys/
                    logs/
                    mediacache/
                    stats/
                    transcoder/
                vhosts002/
                    ...
                vhosts003/
                    ...
                e.t.c

Suggested Update of VHosts.xml

        <?xml version="1.0" encoding="UTF-8"?>
        <Root version="1">
            <VHosts>
                <VHost>
                        <Name>_defaultVHost_</Name>
                        <ConfigDir>${com.wowza.wms.ConfigHome}/vhosts/vhost001/</ConfigDir>
                        <ConnectionLimit>0</ConnectionLimit>
                </VHost> 
                <VHost>
                    <Name>Vhosts002</Name>
                    <ConfigDir>${com.wowza.wms.ConfigHome}/vhosts/vhost002/</ConfigDir>
                    <ConnectionLimit>0</ConnectionLimit>
                </VHost>
                <VHost>
                    <Name>Vhosts003</Name>
                    <ConfigDir>${com.wowza.wms.ConfigHome}/vhosts/vhost003/</ConfigDir>
                    <ConnectionLimit>0</ConnectionLimit>
                </VHost>
            </VHosts>
        </Root>

Suggested Update to: log4j.properties
Note: I have chosent to show all logging options enabled here using the suggested new vHost Layout, as you will see each vHost will internaly store its own global vHost and application log files as well as the core server storing a global server log file.

This helps to keep everything neat and tidy, as well as making it easyer to manage, additionally i have shortened the log filenames, adding the prefix “wowzastreamingengine_” in my eyes is just unnecessary.

log4j.rootCategory=INFO, stdout, serverAccess, serverError, serverStats

# The logging context is
#log4j.logger.[vhost].[application].[appInstance]

# Field list
#date,time,tz,x-event,x-category,x-severity,x-status,x-ctx,x-comment,x-vhost,x-app,x-appinst,x-duration,s-ip,s-port,s-uri,c-ip,c-proto,c-referrer,c-user-agent,c-client-id,cs-bytes,sc-bytes,x-stream-id,x-spos,cs-stream-bytes,sc-stream-bytes,x-sname,x-sname-query,x-file-name,x-file-ext,x-file-size,x-file-length,x-suri,x-suri-stem,x-suri-query,cs-uri-stem,cs-uri-query

# Category list
#server,vhost,application,session,stream,rtsp

# Event list
#connect-pending,connect,disconnect,publish,unpublish,play,pause,setbuffertime,create,destroy,setstreamtype,unpause,seek,stop,record,recordstop,server-start,server-stop,vhost-start,vhost-stop,app-start,app-stop,comment,announce

# To force UTF-8 encoding of log values add the following property to the appender definition (where [appender] is the name of the appender such as "stdout" or "R")
#log4j.appender.[appender].encoding=UTF-8

# Console appender
log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.layout=com.wowza.wms.logging.ECLFPatternLayout
log4j.appender.stdout.layout.Fields=x-severity,x-category,x-event,x-ctx,x-comment
log4j.appender.stdout.layout.OutputHeader=false
log4j.appender.stdout.layout.QuoteFields=false
log4j.appender.stdout.layout.Delimiter=space

# Access appender
log4j.appender.serverAccess=org.apache.log4j.DailyRollingFileAppender
log4j.appender.serverAccess.encoding=UTF-8
log4j.appender.serverAccess.DatePattern='.'yyyy-MM-dd
log4j.appender.serverAccess.File=${com.wowza.wms.ConfigHome}/logs/access.log
log4j.appender.serverAccess.layout=com.wowza.wms.logging.ECLFPatternLayout
#log4j.appender.serverAccess.layout.Fields=x-severity,x-category,x-event;date,time,c-client-id,c-ip,c-port,cs-bytes,sc-bytes,x-duration,x-sname,x-stream-id,x-spos,sc-stream-bytes,cs-stream-bytes,x-file-size,x-file-length,x-ctx,x-comment
log4j.appender.serverAccess.layout.Fields=date,time,tz,x-event,x-category,x-severity,x-status,x-ctx,x-comment,x-vhost,x-app,x-appinst,x-duration,s-ip,s-port,s-uri,c-ip,c-proto,c-referrer,c-user-agent,c-client-id,cs-bytes,sc-bytes,x-stream-id,x-spos,cs-stream-bytes,sc-stream-bytes,x-sname,x-sname-query,x-file-name,x-file-ext,x-file-size,x-file-length,x-suri,x-suri-stem,x-suri-query,cs-uri-stem,cs-uri-query
log4j.appender.serverAccess.layout.OutputHeader=true
log4j.appender.serverAccess.layout.QuoteFields=false
log4j.appender.serverAccess.layout.Delimeter=tab

# Access appender (UDP) - uncomment and add to rootCategory list on first line
#log4j.appender.serverAccessUDP=com.wowza.wms.logging.UDPAppender
#log4j.appender.serverAccessUDP.remoteHost=192.168.15.255
#log4j.appender.serverAccessUDP.port=8881
#log4j.appender.serverAccessUDP.layout=com.wowza.wms.logging.ECLFPatternLayout
#log4j.appender.serverAccessUDP.layout.Fields=x-severity,x-category,x-event;date,time,c-client-id,c-ip,c-port,cs-bytes,sc-bytes,x-duration,x-sname,x-stream-id,x-spos,sc-stream-bytes,cs-stream-bytes,x-file-size,x-file-length,x-ctx,x-comment
#log4j.appender.serverAccessUDP.layout.OutputHeader=true
#log4j.appender.serverAccessUDP.layout.QuoteFields=false
#log4j.appender.serverAccessUDP.layout.Delimeter=tab

# Error appender
log4j.appender.serverError=org.apache.log4j.DailyRollingFileAppender
log4j.appender.serverError.encoding=UTF-8
log4j.appender.serverError.DatePattern='.'yyyy-MM-dd
log4j.appender.serverError.File=${com.wowza.wms.ConfigHome}/logs/error.log
log4j.appender.serverError.layout=com.wowza.wms.logging.ECLFPatternLayout
log4j.appender.serverError.layout.Fields=x-severity,x-category,x-event;date,time,c-client-id,c-ip,c-port,cs-bytes,sc-bytes,x-duration,x-sname,x-stream-id,x-spos,sc-stream-bytes,cs-stream-bytes,x-file-size,x-file-length,x-ctx,x-comment
log4j.appender.serverError.layout.OutputHeader=true
log4j.appender.serverError.layout.QuoteFields=false
log4j.appender.serverError.layout.Delimeter=tab
log4j.appender.serverError.Threshold=WARN

# Statistics appender (to use this appender add "serverStats" to the list of appenders in the first line of this file)
log4j.appender.serverStats=org.apache.log4j.DailyRollingFileAppender
log4j.appender.serverStats.encoding=UTF-8
log4j.appender.serverStats.DatePattern='.'yyyy-MM-dd
log4j.appender.serverStats.File=${com.wowza.wms.ConfigHome}/logs/stats.log
log4j.appender.serverStats.layout=com.wowza.wms.logging.ECLFPatternLayout
log4j.appender.serverStats.layout.Fields=x-severity,x-category,x-event;date,time,c-client-id,c-ip,c-port,cs-bytes,sc-bytes,x-duration,x-sname,x-stream-id,x-spos,sc-stream-bytes,cs-stream-bytes,x-file-size,x-file-length,x-ctx,x-comment
log4j.appender.serverStats.layout.OutputHeader=true
log4j.appender.serverStats.layout.QuoteFields=false
log4j.appender.serverStats.layout.Delimeter=tab
log4j.appender.serverStats.layout.CategoryInclude=session,stream
log4j.appender.serverStats.layout.EventExclude=comment

# Below are logging definitions for dynamic log file generation on a per application basis.
# To use these logging appender, uncomment each of the lines below. It will generate log files
# using the following directory/file structure: 
#
#   [install-dir]/vhosts/[vhost]/logs/[application]/access.log
#   [install-dir]/vhosts/[vhost]/logs/[application]/error.log
#   [install-dir]/vhosts/[vhost]/logs/[application]/stats.log

#### APPLICATION LEVEL LOGGING CONFIG - START ####
log4j.logger.${com.wowza.wms.context.VHost}.${com.wowza.wms.context.Application}=INFO, ${com.wowza.wms.context.VHost}_${com.wowza.wms.context.Application}_access, ${com.wowza.wms.context.VHost}_${com.wowza.wms.context.Application}_error, ${com.wowza.wms.context.VHost}_${com.wowza.wms.context.Application}_stats

log4j.appender.${com.wowza.wms.context.VHost}_${com.wowza.wms.context.Application}_access=org.apache.log4j.DailyRollingFileAppender
log4j.appender.${com.wowza.wms.context.VHost}_${com.wowza.wms.context.Application}_access.DatePattern='.'yyyy-MM-dd
log4j.appender.${com.wowza.wms.context.VHost}_${com.wowza.wms.context.Application}_access.File=${com.wowza.wms.context.VHostConfigHome}/logs/${com.wowza.wms.context.Application}/access.log
log4j.appender.${com.wowza.wms.context.VHost}_${com.wowza.wms.context.Application}_access.layout=com.wowza.wms.logging.ECLFPatternLayout
log4j.appender.${com.wowza.wms.context.VHost}_${com.wowza.wms.context.Application}_access.layout.Fields=x-severity,x-category,x-event;date,time,c-client-id,c-ip,c-port,cs-bytes,sc-bytes,x-duration,x-sname,x-stream-id,x-spos,sc-stream-bytes,cs-stream-bytes,x-file-size,x-file-length,x-ctx,x-comment
log4j.appender.${com.wowza.wms.context.VHost}_${com.wowza.wms.context.Application}_access.layout.OutputHeader=true
log4j.appender.${com.wowza.wms.context.VHost}_${com.wowza.wms.context.Application}_access.layout.QuoteFields=false
log4j.appender.${com.wowza.wms.context.VHost}_${com.wowza.wms.context.Application}_access.layout.Delimeter=tab

log4j.appender.${com.wowza.wms.context.VHost}_${com.wowza.wms.context.Application}_error=org.apache.log4j.DailyRollingFileAppender
log4j.appender.${com.wowza.wms.context.VHost}_${com.wowza.wms.context.Application}_error.DatePattern='.'yyyy-MM-dd
log4j.appender.${com.wowza.wms.context.VHost}_${com.wowza.wms.context.Application}_error.File=${com.wowza.wms.context.VHostConfigHome}/logs/${com.wowza.wms.context.Application}/error.log
log4j.appender.${com.wowza.wms.context.VHost}_${com.wowza.wms.context.Application}_error.layout=com.wowza.wms.logging.ECLFPatternLayout
log4j.appender.${com.wowza.wms.context.VHost}_${com.wowza.wms.context.Application}_error.layout.Fields=x-severity,x-category,x-event;date,time,c-client-id,c-ip,c-port,cs-bytes,sc-bytes,x-duration,x-sname,x-stream-id,x-spos,sc-stream-bytes,cs-stream-bytes,x-file-size,x-file-length,x-ctx,x-comment
log4j.appender.${com.wowza.wms.context.VHost}_${com.wowza.wms.context.Application}_error.layout.OutputHeader=true
log4j.appender.${com.wowza.wms.context.VHost}_${com.wowza.wms.context.Application}_error.layout.QuoteFields=false
log4j.appender.${com.wowza.wms.context.VHost}_${com.wowza.wms.context.Application}_error.layout.Delimeter=tab
log4j.appender.${com.wowza.wms.context.VHost}_${com.wowza.wms.context.Application}_error.Threshold=WARN

log4j.appender.${com.wowza.wms.context.VHost}_${com.wowza.wms.context.Application}_stats=org.apache.log4j.DailyRollingFileAppender
log4j.appender.${com.wowza.wms.context.VHost}_${com.wowza.wms.context.Application}_stats.DatePattern='.'yyyy-MM-dd
log4j.appender.${com.wowza.wms.context.VHost}_${com.wowza.wms.context.Application}_stats.File=${com.wowza.wms.context.VHostConfigHome}/logs/${com.wowza.wms.context.Application}/stats.log
log4j.appender.${com.wowza.wms.context.VHost}_${com.wowza.wms.context.Application}_stats.layout=com.wowza.wms.logging.ECLFPatternLayout
log4j.appender.${com.wowza.wms.context.VHost}_${com.wowza.wms.context.Application}_stats.layout.Fields=x-severity,x-category,x-event;date,time,c-client-id,c-ip,c-port,cs-bytes,sc-bytes,x-duration,x-sname,x-stream-id,x-spos,sc-stream-bytes,cs-stream-bytes,x-file-size,x-file-length,x-ctx,x-comment
log4j.appender.${com.wowza.wms.context.VHost}_${com.wowza.wms.context.Application}_stats.layout.OutputHeader=true
log4j.appender.${com.wowza.wms.context.VHost}_${com.wowza.wms.context.Application}_stats.layout.QuoteFields=false
log4j.appender.${com.wowza.wms.context.VHost}_${com.wowza.wms.context.Application}_stats.layout.Delimeter=tab
log4j.appender.${com.wowza.wms.context.VHost}_${com.wowza.wms.context.Application}_stats.layout.CategoryInclude=session,stream
log4j.appender.${com.wowza.wms.context.VHost}_${com.wowza.wms.context.Application}_stats.layout.EventExclude=comment
#### APPLICATION LEVEL LOGGING CONFIG - STOP ####

# Below are logging definitions for dynamic log file generation on a per virtual host basis.
# To use these logging appender, uncomment each of the lines below. It will generate log files
# using the following directory/file structure: 
#
#   [install-dir]/vhosts/[vhost]/logs/access.log
#   [install-dir]/vhosts/[vhost]/logs/error.log
#   [install-dir]/vhosts/[vhost]/logs/stats.log

#### VHOST LEVEL LOGGING CONFIG - START ####
log4j.logger.${com.wowza.wms.context.VHost}=INFO, ${com.wowza.wms.context.VHost}_access, ${com.wowza.wms.context.VHost}_error, ${com.wowza.wms.context.VHost}_stats

log4j.appender.${com.wowza.wms.context.VHost}_access=org.apache.log4j.DailyRollingFileAppender
log4j.appender.${com.wowza.wms.context.VHost}_access.DatePattern='.'yyyy-MM-dd
log4j.appender.${com.wowza.wms.context.VHost}_access.File=${com.wowza.wms.context.VHostConfigHome}/logs/access.log
log4j.appender.${com.wowza.wms.context.VHost}_access.layout=com.wowza.wms.logging.ECLFPatternLayout
log4j.appender.${com.wowza.wms.context.VHost}_access.layout.Fields=x-severity,x-category,x-event;date,time,c-client-id,c-ip,c-port,cs-bytes,sc-bytes,x-duration,x-sname,x-stream-id,x-spos,sc-stream-bytes,cs-stream-bytes,x-file-size,x-file-length,x-ctx,x-comment
log4j.appender.${com.wowza.wms.context.VHost}_access.layout.OutputHeader=true
log4j.appender.${com.wowza.wms.context.VHost}_access.layout.QuoteFields=false
log4j.appender.${com.wowza.wms.context.VHost}_access.layout.Delimeter=tab

log4j.appender.${com.wowza.wms.context.VHost}_error=org.apache.log4j.DailyRollingFileAppender
log4j.appender.${com.wowza.wms.context.VHost}_error.DatePattern='.'yyyy-MM-dd
log4j.appender.${com.wowza.wms.context.VHost}_error.File=${com.wowza.wms.context.VHostConfigHome}/logs/error.log
log4j.appender.${com.wowza.wms.context.VHost}_error.layout=com.wowza.wms.logging.ECLFPatternLayout
log4j.appender.${com.wowza.wms.context.VHost}_error.layout.Fields=x-severity,x-category,x-event;date,time,c-client-id,c-ip,c-port,cs-bytes,sc-bytes,x-duration,x-sname,x-stream-id,x-spos,sc-stream-bytes,cs-stream-bytes,x-file-size,x-file-length,x-ctx,x-comment
log4j.appender.${com.wowza.wms.context.VHost}_error.layout.OutputHeader=true
log4j.appender.${com.wowza.wms.context.VHost}_error.layout.QuoteFields=false
log4j.appender.${com.wowza.wms.context.VHost}_error.layout.Delimeter=tab
log4j.appender.${com.wowza.wms.context.VHost}_error.Threshold=WARN

log4j.appender.${com.wowza.wms.context.VHost}_stats=org.apache.log4j.DailyRollingFileAppender
log4j.appender.${com.wowza.wms.context.VHost}_stats.DatePattern='.'yyyy-MM-dd
log4j.appender.${com.wowza.wms.context.VHost}_stats.File=${com.wowza.wms.context.VHostConfigHome}/logs/stats.log
log4j.appender.${com.wowza.wms.context.VHost}_stats.layout=com.wowza.wms.logging.ECLFPatternLayout
log4j.appender.${com.wowza.wms.context.VHost}_stats.layout.Fields=x-severity,x-category,x-event;date,time,c-client-id,c-ip,c-port,cs-bytes,sc-bytes,x-duration,x-sname,x-stream-id,x-spos,sc-stream-bytes,cs-stream-bytes,x-file-size,x-file-length,x-ctx,x-comment
log4j.appender.${com.wowza.wms.context.VHost}_stats.layout.OutputHeader=true
log4j.appender.${com.wowza.wms.context.VHost}_stats.layout.QuoteFields=false
log4j.appender.${com.wowza.wms.context.VHost}_stats.layout.Delimeter=tab
log4j.appender.${com.wowza.wms.context.VHost}_stats.layout.CategoryInclude=session,stream
log4j.appender.${com.wowza.wms.context.VHost}_stats.layout.EventExclude=comment
#### VHOST LEVEL LOGGING CONFIG - STOP ####

Additional Food for Thought

A way to login an administrate one vHost per time insted of logging in to the global manager (please excuse if this is already possible, iv yet to come across how to achieve this.)
As requested in the previous suggestion thread, a full suite of API’s to add/remove/manage vHosts on the fly with out the need for rebooting the server. e.g dynamically up datable configuration at runtime.

Thanks for taking the time to read this,

If there are any questions, further suggestions or general comments relating to any aspect of this suggestion please ask/post them!

Lee.

James_Sherry · January 20, 2015, 12:29pm

Hi Lee,

Thank you very much for your input. We will review and let you know what happens next.

Thanks,

-Jamie

Lee_Wickham · January 22, 2015, 7:37pm

Hi Lee,

Thank you very much for your input. We will review and let you know what happens next.

Thanks,

-Jamie

Your welcome Jamie, I look forward to hearing what comes out of the review process xD

Topic		Replies	Views
Request/Suggestion: Reload configuration files on the fly with out restarting. Wowza Streaming Engine	2	2356	December 22, 2014
Vhosts.xml and vhost.xml Wowza Streaming Engine	2	3615	September 19, 2007
Changing Wowza configuration path Wowza Streaming Engine	7	3492	May 21, 2015
startup.xml Wowza Streaming Engine	4	2716	March 24, 2010
Missing xvhost of ServerListenerStreamPublisher in ACCESS LOG Wowza Streaming Engine server-administration	3	2718	May 19, 2016

Suggestion/Request: A restructure of Wowza'a directory Layout (vhosts spersific)

Related topics