Logging and monitoring

Hi,

As a hosting provider I would really like to see a Flash based streaming server that provides logging and monitoring in line with Helix/Windows Media Server.

Flashcom 1.5 was useless. Flash Media Server 2 is much better at the monitoring, although the logging still doesn’t seem right (although I have not spent much time looking at it).

My requirements are really for a cut down version of flash streaming server. Not that many UK clients are clued up enough to make use of the interactive features available and just want straight hosting of their videos, but they do want to know how many times their videos were watched, and I need to know how much data was transferred as that’s how we do our billing.

In an ideal world I would like to see a monitoring tool which will show me which streams are currently playing, total bandwidth utilisation at a given time

and to make things easy CPU and Memory usage so that I can upgrade euipment if necessary. For the logging all I really want is a single line appended to the log file for each stream played in W3C complient format stating date and time of play and data transfer so that I can run it though a stats package. A carriage return at the end of each line is also useful so that you can scan through it manually if you need to :slight_smile:

Thanks for the feedback and suggestions. We are will working on monitoring. We are thinking about providing serveral different interfaces (SOAP, HTML, SNMP, RPC) into a common API for monitoring the server. Our thoughts are the same as far as keeping it simple. Is the server up? Simple stats on usage and resource consumption.

The logging suggestions are good as well. Some of the logging we have in place. We are logging using the java log4j package which provides a lot of configuration flexibility. The logs are being saved in the /logs folder in the installation folder. The configuration is in the conf/log4j.properties file. We are still working on logging and what you describe should be in place by the time we ship the 1.0 product.

Thanks for taking the time to send us your suggestions.

Charlie

The only real documentation down to the field level is to map the objects to their definition in the server API. Some of the fields are documented there. I realize this is not a great answer. But it what we have at this time.

Charlie

We certainly feel the need and are actively working on it. We do not have a definitive date as to when the first release will be available.

Charlie

See these forum posts:

http://community.wowza.com/t/-/60

http://community.wowza.com/t/-/84

http://community.wowza.com/t/-/236

Charlie

Thanks for the feedback and choosing Wowza Pro. I agree with all of your points and these are areas where we need to improve. We do see the gaping hole in our administrative tools. JConsole is certainly a lot better than MC4J so for now you may want to focus your efforts there. Using a general purpose tool like JConsole for server monitoring and management has its pluses and minuses but providing a more focused Wowza Pro specific tool is a necessity. We are working on this.

I am not sure I understand the issue with logging. We modeled our logging after the Flash Media Server logging mechanism. For the most part the file format, down to the field names, is exactly the same. Wowza Pro provides W3C compliant logs that can be consumed by most logging analysis programs that support streaming servers. So please describe in more detail where there is a deficiency. That being said there is a lot more we need to do with logging and log analysis. Again, something we plan on focusing in future versions of Wowza Pro. It is tough to get it all done at once.

So thanks for the feeback. We are a small focused company that is working hard to provide the best streaming server in the industry. It is important to get the fundamentals right.

Charlie

I am not sure why JConsole would not work. Probably for the same reason MC4j will not work. If you want to contact me offlist and send me your conf/Server.xml file I can take a look at it (charlie@wowza.com).

Charlie

Which link is broken?

Charlie

As a Wowza user I would like to add to the discussion that using jconsole or mc4j might seem more complex at first but will give you way more information about what is actually going on. It will also give you more options to control the server as well. Compared to the FMS admin things might look more complicated but with some practice it will proof to be more efficient. I do understand the need of a plain and simple admin page that will show the basic stats like the FMS admin does.

About logging. There is a difference between logging application stats or application errors and logging system errors/warnings that should end up in syslog. In my experience Wowza can log statistics, errors and more per server, vhost, or application instance any way you like.

I updated it.

The ones that I replaced had been deleted.

Richard

Just to tie things together so it doesn’t seem like a loose end, I’m going to reference your other thread:

https://www.wowza.com/forums/showthread.php?p=43368#1

Richard

I’m gonna resurrect this discussion and agree with the OP regarding a monitoring system that’s admin-friendly vs. developer-friendly. I’m in the same boat as chasho, and have very little interest in becoming a Java dev. My brain is full enough dealing with all the information involved in streaming.

If anyone in development world wants to make something that’s got a great web interface that you connect to on the server, that would be awesome.

Stuff that’s of interest to me as a content publisher/administrator:

  • current/peak bandwidth consumption

  • current/peak connection count (overall)

  • current/peak connection count (per stream)

  • streams/applications currently configured and published (possibly with a link to monitor that stream)

In origin/edge configuration, add:

  • connected servers and which streams they’re pulling

  • aggregate counts of the above

Having the kind of easy admin panel that you find with many CDNs would be of tremendous value.

Charlie:

Excuse the reply to such an old post, but it hits on a sore spot for me that I’d like to see if there’s current feedback on. Hopefully my comments are seen as constructive criticism. :slight_smile:

I just went through a testing period with Wowza and FMS, and chose Wowza because on a feature / price comparison, it wins, and it did perform slightly better in my configuration than FMS. So far, it is running well and doing what is needed, without too much installation or troubleshooting difficulty. Your performance hints in the forum have been most helpful in troubleshooting the issues I have experienced.

The problem I have is the complete lack of tools available for me as an administrator. I stress that I am an administrator, NOT a Java developer, XML developer, SOAP developer, etc. I can do shell scripting, I can write a batch file, but beyond that, I have neither the time, skills, nor interest in having to build an interface for a server to manage it and understand what it is doing. I use tools that someone else has built to do this kind of work, and I even pay for them (sometimes).

There needs to be a management / monitoring client built for this product similar to what FMS has, plain and simple. MC4J hasn’t had decent dev work done on it in years (and isn’t stable in my experience), the Jconsole apps require me to have a deeper understanding of Java than I want, much less have, and not knowing what the heck is going on, per stream, per VHost, just stinks when you’re trying to troubleshoot a problem. I don’t mind working with Open Source tools that maybe don’t have the pretty interface of something like FMS’ admin console, but I need it to be stable, and I need it to be documented in Admin-speak, not Developer-speak. None of the recommended tools meets these needs. I don’t need an IDE. I’m not a developer. I’m not building custom modules. I’ll let my developers do that, who do need an IDE. I need an administration console that provides me real time performance information, since this is such a performance sensitive environment.

If Wowza isn’t going to spend the cycles / $ to develop their own admin console, then they should alternatively spend those cycles documenting how to make these other tools provide the most important information for performance monitoring / tuning and overall system administration, without assuming the user is a Java developer and understands the intricacies of classes, Mbeans, JVM trace calls, etc. This was a major issue that almost swung my decision to FMS away from Wowza. Their documents are abundant, and they approach things from a system admin perspective in the user guides, install guides, performance tuning guides, and from a developer’s perspective in the API guides, etc. I understand the resource pool is larger at Adobe, and Wowza has grown quickly with a single User Guide document laid out the way it currently is and without a home grown console tool, but to take things to the next level, I think this is going to be important. Given what I know now, it may become an even more important issue when evaluating these two tools for my next client.

If I’m missing some documentation somewhere that makes using MC4J more stable and lays out how to get the most out of it, point me to it. I haven’t been able to find it, though.

I would also strongly suggest a move away from the current logging practice. I support a lot of servers doing various tasks, and you are the only application in use in my supported farm that doesn’t have some way to integrate into syslog. Unacceptable. The java logging mechanism you use may be great, but the fact that I am having to figure out these logs to get any relevant information out of them is just ridiculous. The documentation stinks, quite frankly, and pointing me to someone else to figure out how to log events from YOUR application, is not an Enterprise class approach. It is shifting the buck. Again, FMS does it the right way. They use standard syslog mechanisms to get you all the information you need. Of course, we could debate whether this is the ‘right way’ or not, but you can’t really argue that it is the ‘standard way’, and if you want better market penetration, falling into a standardized way of doing things helps you. System administrators don’t want multiple log mechanisms. It makes it harder to correlate problems when you have them.

I do appreciate the articles in the forum about using AWstats with your logging mechanism. That will be helpful as I move to the next phase of the implementation.

Of course, these are just one guy’s thoughts. Take them for what they’re worth. You’ve built an app that seems tuned well (despite it being java based), has been stable thus far in my environment, and seems secure. Kudos, and thanks.

Charlie

Thanks for replying. I forgot to add that one other reason I went with Wowza was, while Adobe has more documents, you personally seem to care a lot more about customer experience and take the time to reply. That means a TON.

I took a look at JConsole, and after half an hour of beating my head against the desk just trying to get a connection, gave up. At least MC4J would connect (although it isn’t now).

I don’t mind RTFMing. Is there a good HowTo or other doc you’d recommend using that explains what JConsole really does and how it does it, that doesn’t rely on my being a Java developer? I haven’t been able to find anything that makes it make a licking bit of sense. I’m really not a noob to sys administration, but this has created a whole new class of frustration for me.

As to logging, I completely understand the difference between app logs and system logs. However, Apache HTTPD (an application) does a very nice job of utilizing syslog functionality and form to produce immediately understandable logs, without my having to configure anything. I’ve learned over the years that there are things I want to add to the logs, and because it uses standard nomenclature, that wasn’t hard to do. Perhaps it just comes back to documentation and explanation of the logging functions. Perhaps it’s just that everything having to do with setting up the logs looks like I’m trying to build a Java class. I’m not sure. I just find the logs incredibly difficult to read and determine what the problem is, and I feel like I’m having to experiment with the log settings way too much. I rely on logs to troubleshoot things, and I don’t feel confident that the information is in the error log to help me do that.

Thanks!!!

Thanks Charlie. I just now got it working by using my internal management network that I have VPN access to, and it seems to be working fine now. I’m not precisely sure what change I made caused it to work, but it’s giving me a connection now.

Is there a reference doc on all the attributes contained in the Mbeans and what they really mean? I get total connections, but some of the names are foreign to me, like messagesLossCountRate. It’s clearly b/c I’m new to streaming, but I could probably define that several ways, and knowing exactly what I’m looking at is crucial to tuning.

I would really like to be able to collect live data. FMS allows us to grab stats in an xml fashion from thier admin server. Does Wowza have anything similar? Any plans? Any timelines. This is really important to us as 80% of my usage is live.

Hi, weatherbytech,

How do you configure Cacti to log only the Wowza variables? From what little I’ve seen of Cacti, it uses SNMPD and will get system-wide values. But if there are several apps running on the server, however minor in comparison to Wowza, the results will be contaminated.

What am I missing?

I’d like to chip in my comments on JConsole and monitoring. I have leafed through all the objects that can be inspected with JConsole, and there’s a wealth of information there.

The problem is that as far as I can tell, you can only inspect basic type properties like strings and ints. It would be great to be able to inspect the properties of SharedObjects or the Properties of a Client (you can in fact run toString() on the Properties, but it’s rather inefficient to wade through the string representation of the whole WMSProperties object). But JConsole doesn’t handle ‘mixed’ variables like AMFData.

If you want to debug a multi-user interactive application that heavily uses shared objects, this is crucial to be able to see what’s happening.

Regards,

Sander Kruger

The link for Wowza 2.0 gives an error message. It states that I do not have permission to access the page.

I’m gonna resurrect this discussion and agree with the OP regarding a monitoring system that’s admin-friendly vs. developer-friendly. I’m in the same boat as chasho, and have very little interest in becoming a Java dev. My brain is full enough dealing with all the information involved in streaming.

If anyone in development world wants to make something that’s got a great web interface that you connect to on the server, that would be awesome.

Stuff that’s of interest to me as a content publisher/administrator:

  • current/peak bandwidth consumption

  • current/peak connection count (overall)

  • current/peak connection count (per stream)

  • streams/applications currently configured and published (possibly with a link to monitor that stream)

In origin/edge configuration, add:

  • connected servers and which streams they’re pulling

  • aggregate counts of the above

Having the kind of easy admin panel that you find with many CDNs would be of tremendous value.

Why would you not just use CACTI? Very clean graphs and easy to configure…