Can anyone recommend a log analyzer that will process Wowza/FMS logs?
Thanks,
Penneths
I use awstats:
I have spent quite a lot of time on this, and Charlie really helped out when creating Wowza’s logs in the first place. I’m not sure what FCS2 is like, but 1.5 consisted of some weird ascii dump which was impossible to work with…
I really only wanted to know which files had been played and the amount of data transfer for on-demand files which I have got working really well (see below). It will still work for live stuff. As for analysing more interactive stuff, that will be a bit more difficult.
Anyway here is what I do:
I edit the log4jproperties file as follows:
log4j.appender.R=org.apache.log4j.DailyRollingFileAppender
log4j.appender.R.DatePattern='.'yyyy-MM-dd
log4j.appender.R.File=D:/logs/wowza/wowzamediaserver_access.log
log4j.appender.R.layout=com.wowza.wms.logging.ECLFPatternLayout
log4j.appender.R.layout.OutputHeader=true
log4j.appender.R.layout.QuoteFields=false
log4j.appender.R.layout.Delimeter=tab
log4j.appender.R.layout.Fields=c-ip,date,time,x-sname,x-status,c-proto,sc-stream-bytes
log4j.appender.R.layout.CategoryInclude=stream
log4j.appender.R.layout.EventInclude=destroy
I am only logging the fields that I specifically want, and only doing so for the category stream at the point when it is destroyed. This provides us with a simple line in the log file with the important information:
192.168.1.70 2007-03-08 18:54:06 demo/merry_melodies_falling_hare 200 rtmpt (HTTP-1.1)
I have been streaming via rtmpt over port 80 to get around firewalls and the space in rtmpt (HTTP-1.1) may cause an issue if you do not use a tab as the log delimiter, so best keep it as that!
Next I install awstats. this will require a Perl interpreter (If you are using Windows then I recommend ActivePerl).
You then need to set up a config file for awstats. (Copy and paste the demo one and edit it). The important bits are:
LogType=S
LogFormat=“c-ip date time cs-uri-stem c-status protocol sc-bytes”
ValidHTTPCodes=“200 304 408”
(I add 408 so that interrupted streams still log the data transfer)
And if you are analysing multiple file on a Windows server then
LogFile=““C:/Perl/bin/perl.exe” “d:/Program Files/AWStats/tools/logresolvemerge.pl” d:/logs/wowza/* |”
The double quotes took me ages to work out!!!
Finally you will need to edit the awstats.pl file as it does not know what to do with the rtmp(t) protocols and throws up errors unless you include them as options in line 6338.
Just add the extra OR options:
|| $field[$pos_method] eq ‘rtmp’ || $field[$pos_method] eq ‘rtmpt (HTTP-1.1)’
Assuming you’ve set up your cgi-bin properly and filled any little gaps that I may have left, you should be able to get your stats by going to:
http:///cgi-bin/awstats.pl?config=
Hit the update link and you will have instant stats!!!
I have spent quite a lot of time editing the config file as there are lots of irrelevant bits in there, and am really happy with the results.
Hope that all makes sense - it did take me a good week to sort it all out!
I have not done any work with awstats. Hopefully someone else on the forums can help.
Charlie
I can’t remember what the c-proto field is, but it looks like that’s where your problem is.
You need to make sure that your logfiles from Wowza and the “Logformat” string are consistent.
I remember encountering similar problems, but stripped out everything that was not important and substituted some “dummy” variables just to keep awstats happy, and then added in the important data that I needed once that was working…
Hi Chaso,
Yes that does sound familiar, although it’s over a year ago and I can’t remember exactly what I did to fix it. Have you followed the instructions that I set up?
Looking at it, it’s probably because awstats does not recognise rtmp or rtmpt by default so you will have to add those into the awstats.pl line. It was line 6338 a year ago, but I expect this has changed with a newer version.
Look for $field[$pos_method] and add the extra OR clauses:
|| $field[$pos_method] eq ‘rtmp’ || $field[$pos_method] eq ‘rtmpt (HTTP-1.1)’
Jim
Jim,
Many thanks for your help. So far I’m impressed by Wowza and would rather not use FMS. I’ll follow all the steps in your post and see if I can get it to work.
Thanks again,
Penneths
This line is not in the access appender, try commenting it out:
log4j.appender.serverStats.layout.CategoryInclude=session,stream
Richard
You can modify your Fields list. You have this:
log4j.appender.${com.wowza.wms.context.VHost}_${com.wowza.wms.context.Application}_stats.layout.Fields=x-severity,x-category,x-event;date,time,c-client-id,c-ip,c-port,cs-bytes,sc-bytes,x-duration,x-sname,x-stream-id,x-spos,sc-strea
m-bytes,cs-stream-bytes,x-file-size,x-file-length,x-ctx,x-comment
The superset of available log Fields is in the log4j.properties file comment:
# Field list
#date,time,tz,x-event,x-category,x-severity,x-status,x-ctx,x-comment,x-vhost,x-app,x-appinst,x-duration,s-ip,s-port,s-uri,c-ip,c-proto,c-referrer,c-user-agent,c-client-id,cs-bytes,sc-bytes,x-stream-id,x-spos,cs-stream-bytes,sc-stream-bytes,x-sname,x-sname-query,x-file-name,x-file-ext,x-file-size,x-file-length,x-suri,x-suri-stem,x-suri-query,cs-uri-stem,cs-uri-query
Richard
Please tell us what you are trying to do, and what does not work.
Richard
Here is an article on using AWStats and Wowza:
http://www.wmconsulting.info/awstats-mod-for-process-wowza-stats/
And there is some discussion here that might be useful that is easy to find if you search for “Wowza AWStats”
Richard
I have problems with this setup. My problem is with editing the awstats.pl file.
I’m not sure if I set the line specifying fields inside log file:
|| $field[$pos_method] eq ‘rtmp’ || $field[$pos_method] eq ‘rtmpt (HTTP-1.1)’
when I run perl update, all of my lines inside log files get dropped:
perl awstats.pl -config=test2 -update -showdropped
So I get the next response:
C:\Inetpub\SPS_stat\cgi-bin>perl awstats.pl -config=test2 -update -showdropped
Create/Update database for config “./awstats.test2.conf” by AWStats version 6.7
(build 1.892)
From data in log file "C:\Documents and Settings\jurera\Desktop\testni_wowza.log
"…
Phase 1 : First bypass old records, searching new record…
Searching new records from beginning of log file…
Dropped record (method/protocol ‘c-proto’ not qualified when LogType=S): #Fields
: c-ip date time x-sname x-status c-proto sc-stream-bytes
Jumped lines in file: 0
Parsed lines in file: 5
Found 1 dropped records,
Found 4 corrupted records,
Found 0 old records,
Found 0 new qualified records.
Has anybody come up with the same problem?
Hi,
Great info. I am already familair with AWStats, so that helps. The problem that is here going on are two things.
- The logfile in wowza is tab delimeted, while AWStats defaults thinks that is space delimeted. So you have to edit the settings LogSeparator to ‘\t’
That wil fix the corrupted records.
- The problem that then rises is that when you use rtmp streaming, it is not a valid protocol. AWstats will reject all lines with the rtmp as protocol.
To fix this, you edit the awstats.pl file from AWStats. Go to line +/-6400. There should be the line:
elsif (($LogType eq 'W' || $LogType eq 'S') && ($field[$pos_method] eq 'GET' || $field[$pos_method] eq 'mms' || $field[$pos_method] eq 'rtsp' |
| $field[$pos_method] eq 'http' || $field[$pos_method] eq 'RTP' )) {
add the following to the if statement:
|| $field[$pos_method] eq 'rtmp'
so it will become:
elsif (($LogType eq 'W' || $LogType eq 'S') && ($field[$pos_method] eq 'GET' || $field[$pos_method] eq 'mms' || $field[$pos_method] eq 'rtsp' |
| $field[$pos_method] eq 'http' || $field[$pos_method] eq 'RTP'|| $field[$pos_method] eq 'rtmp')) {
So AWStats will not reject lines with the rtmp protocol. And your done!
There will be one dropped line, and that is the header line which is ok.
I hope this solves the problems and complete this little HOWTO of streamingwizard
TheYOSH
Jim and/or Yosh:
Did Awstats complain to either of you about not having a %method or %methodurl variable in your LogFormat string?
I’m encountering it giving me an error that it needs the %method in the string, but it doesn’t seem to matter where I put it, it doesn’t like it as a dummy variable.
Thanks!
To avoid modifying AWStats, you can use the following line in the config:
LogFormat="cat /path/to/logs/*.log | sed -r 's/rtmp(t \(HTTP-1.1\))?/http/' |"
Hi, fo use AWStats and support all the protocols need to do some changes into the awstats.pl
I have this running with the package modified. I’ll try to post the instruction to do this during the Week.
Regards,
Alejandro
Hi Guys,
Here I have make public the latest mod for Awstats and Wowza Logs, please try this and send me any issue, all the feedback is appreciated…
http://www.wmconsulting.info/awstats-mod-for-process-wowza-stats/
This mod is based into the latest Awstats version with some extra lines for support all the protocols and Live Stream… I’ll add the changelog soon…
Regards,
Ale
what error do you have?
Hi Ban_ban,
the Dropeed lines is normally why your line sintax is not the same than awstats have setting.
add to your line the params: -showdropped
this will show you the dropped lines, after that the unique way to see what happend is compare the awstats parser line with the wowza logs, possible you have some param more or into other order.
Are you using RTMP only or are make the tests with HLS or RTSP?
Paste some lines here I’ll try to help…
Regards,
Alejandro
wmconsulting.info
Minor correct to post about log modified so you don’t have to hack awstats:
Should read LogFile= instead of LogFormat
LogFile="cat /path/to/logs/*.log | sed -r 's/rtmp(t \(HTTP-1.1\))?/http/' |"
I’ve set my logging up differently as follows:
log4j.logger.${com.wowza.wms.context.VHost}.${com.wowza.wms.context.Application}=INFO, ${com.wowza.wms.context.VHost}_${com.wowza.wms.context.Application}_access, ${com.wowza.wms.context.VHost}_${com.wowza.w
ms.context.Application}_error, ${com.wowza.wms.context.VHost}_${com.wowza.wms.context.Application}_stats
log4j.appender.${com.wowza.wms.context.VHost}_${com.wowza.wms.context.Application}_access=org.apache.log4j.DailyRollingFileAppender
log4j.appender.${com.wowza.wms.context.VHost}_${com.wowza.wms.context.Application}_access.DatePattern='.'yyyy-MM-dd
log4j.appender.${com.wowza.wms.context.VHost}_${com.wowza.wms.context.Application}_access.File=${com.wowza.wms.ConfigHome}/logs//${com.wowza.wms.context.Application}/wowzamediaserver_access.log
log4j.appender.${com.wowza.wms.context.VHost}_${com.wowza.wms.context.Application}_access.layout=com.wowza.wms.logging.ECLFPatternLayout
log4j.appender.${com.wowza.wms.context.VHost}_${com.wowza.wms.context.Application}_access.layout.Fields=c-ip,date,time,x-sname,x-status,c-proto,sc-stream-bytes
log4j.appender.${com.wowza.wms.context.VHost}_${com.wowza.wms.context.Application}_access.layout.OutputHeader=true
log4j.appender.${com.wowza.wms.context.VHost}_${com.wowza.wms.context.Application}_access.layout.QuoteFields=false
log4j.appender.${com.wowza.wms.context.VHost}_${com.wowza.wms.context.Application}_access.layout.Delimeter=tab
log4j.appender.${com.wowza.wms.context.VHost}_${com.wowza.wms.context.Application}_access.layout.CategoryInclude=session,stream
log4j.appender.${com.wowza.wms.context.VHost}_${com.wowza.wms.context.Application}_access.layout.EventInclude=destroy
I notice in the main access.log it shows streams from an iphone…
However in the above setup it doesn’t? Can’t seem to work out why any ideas?