MagicStats 2.0 User Guide

Abstract:

MagicStats is a flexible package for managing and analyzing web site statistics. It is capable of being expanded through plugins, user defined themes, and user configuration files. This document gives the mile-high overview of how everything works so that you can use it to its full potential.
  1. MagicStats Overview:
  2. MagicStats Configuration File:
    1. Configuration File Overview
    2. The Global Section
    3. How Variables Really Work...
    4. Theme Sections
  3. MagicStats Command Line Options:
  4. Conclusion:

MagicStats Overview:

The MagicStats system is composed of three "levels" of functionality. In this manual, we discuss the bottom level: The configuration file. The configuration file gives the base line customization features to MagicStats, and is how most users interact with it. We also discuss the command line options that MagicStats recognizes which are useful to know when setting MagicStats up.

Beyond the configuration file, other manuals discuss how to create Themes and how to create plugins for the themes to use. Note that MagicStats was designed with usability in mind. That means that anyone comfortable editing a text file can work on the config file level, anyone comfortable with HTML can work on the Themes level, and one who is comfortable with C++ can work on the Plugins level.

MagicStats Configuration File:

Everyone who uses MagicStats must have a configuration file to tell it what to do. This configuration file tells it where to look for files, how to filter and mangle the access records, and how to spit them out as HTML reports. Here is an example of a configuration file:

#----------------------------------------------------------------------------
[Global]
#------=
#Access Log    	      	: "/usr/local/httpd/logs/access_log";
Access Log		: [ ["/var/log/httpd/access_log.1", "http://www.nondot.org"],
			    ["/var/log/httpd/access_log.2", "http://www.nondot.org"],
			    ["/var/log/httpd/access_log", "http://www.nondot.org"]
			  ];
Data Filename		: "~/.MagicStats/MagicStats.dat";
# Define some global variables...
$BGColor		: "#FFFFFF";
Resources 		: [ ["/var/lib/MagicStats/", "/MagicStatsInst"],
                            ["/home/sabre/public_html/MagicStats/", "/~sabre/MagicStats"]
                          ];
Filters			: [ ["UnEscapeURL" ],
			    ["RemoveIndexFilename", "index.html", 
						    "index.shtml",
						    "index.htm", 
						    "index.cgi", 
						    "/"],
			    ["Domainify", "*", "" ]
			  ];
#----------------------------------------------------------------------------
[[Fancy]]
#-------=
$SectionFilter		: ["*sabre*",""];
$PageFilter		: ($SectionFilter) + "*.GIF;*.JPG;*.gif;*.jpg;*.class";
$PageName		: "Chris's Web Creations";
$OutputPrefix		: "Sabre";
$PathPrefix		: "~/public_html/stats/";
$DailyLink		: ($PathPrefix) + "index.html";
$MonthlyLink		: ($PathPrefix) + "SabreMonthly.html";
#----------------------------------------------------------------------------
[[Fancy]]
#-------=
$PageName		: "MagicStats Web Page";
$SectionFilter		: ["http://www.nondot.org/MagicStats/*",""];
$PageFilter		: ($SectionFilter) + "*.GIF;*.JPG;*.gif;*.jpg;*.class";
$OutputPrefix		: "MS";
$PathPrefix		: "/home/MagicStats/public_html/stats/";
$DailyLink		: ($PathPrefix) + "index.html";
$MonthlyLink		: ($PathPrefix) + "Monthly.html";
We'll be discussing various parts of this standard form configuration file, starting at the top!

Configuration File Overview

The configuration file is a simple structured form for communicating with MagicStats. It consists of a number of sections (which are denoted with square brackets: [..]), statements, variable declarations, and comments. Comments are defined to start with the '#' character and run until the end of line (like shell scripts or perl for example). MagicStats completely ignore whitespace characters for variable or statement names (i.e. "ACCESSLOG" = "ACCeSS Log" = "ACC E SS Lo g").

Each statement or variable declaration is of the form: "something : value ;". By convention, variables start with the dollar sign character ('$'), and statements start with letters. MagicStats completely ignores variables (or rather it just keeps track of them and passes them along), but they are the primary mechanism for communicating preferences to themes.

The Global Section

The [Global] section is used to tell MagicStats how to configure itself. It contains settings that describe how and where to get accesses to process (the Access Log parameter), how to filter accesses (the Filters parameter), and where to find plugins and themes (the Resources parameter). Here's a quick rundown of what these things do:

The AccessLog Parameter

The AccessLog parameter tells MagicStats how to get accesses to process. Like this example, MagicStats can read from multiple different accesslogs at the same time, even if they have overlapping timeperiods (for example each virtual domain may have a different log file). Additionally, if the log file does not contain domain information, you can specify a default domain to use for the whole log file... An optional third parameters for each log is the name of the AccessFormatPlugin to use to parse the log file. MagicStats automatically tries to determine the correct plugin to use, but if it fails, you can override it (to get a list of available plugins, run "MagicStats -P" and take a look!).

The DataFilename Parameter

MagicStats runs in purely incremental mode. This means that MagicStats can be run as frequently as desired without messing up the reports that it generates. To accomplish this, it has to save its state out to a file whenever it is done processing accesses. By using this parameter, you can set the location that you want MagicStats to write its state to. If you don't want to override the default (which is "~/.MagicStats/MagicStats.dat", then you don't have to define it...

The Resources Parameter

This parameter tells MagicStats where to look for "resources". MagicStats considers resources to be plugins and themes that it can use... and it needs to know both the path to these and the URL of these directories on the web server (because themes may use graphics out of their directories for example). Normally you only need to include the directory that MagicStats was installed into (in this case /var/lib/MagicStats, but if you have custom (local) themes or plugins, you can include the directory that they reside under. If you would like to see what one of these directories should look like (it is created by the makefile), you can look here...

The Filters Parameter

The Filters parameter tells MagicStats how to process accesses as they are streamed in from the log files. This sets up a processing chain to take accesses from a relatively raw form to a nice presentable form. Here is what these do, for example:

  1. UnEscapeURL: Transforms escape characters in the URL field of the accesses into the character that it represents (such as %20 into a space character).
  2. RemoveIndexFilename: This plugin strips off the ends off of URLs that end with the specified strings. This is quite useful because references to http://www.nondot.org/~sabre/ and http://www.nondot.org/~sabre/index.html both refer to the same document and should be counted as such.
  3. Domainify: This prepends each URL with the domain of the access log that it comes from. This isn't necessary, I just prefer the "http://" look. On sites with multiple domains being hosted, this is a neccesity, so that links resolve correctly. The two parameters to it are a filter so that only hits that match the filter are "domainified". In this case, I domainify everything, but using a filter of "*", "~*" is useful because it domainifies all hits that don't start with a tilde character.

Other things that are useful are the (currently unimplemented) DNS resolver plugin, and the Rewrite plugin.

How Variables Really Work...

If you really looked at the example, you may have noticed that most of the variables in the example are defined more than one time... what's going on here?

Variables are defined in a "scope" that is the enclosing section of the configuration file. Thus, variables defined in the Global section are active all throughout the global section. Variables defined in the "Fancy:Sabres" section (to be described next) are active in that section. Additionally, variables flow through the following sections until the end of the file, or when they are undefined. This means, for example, that the $BGColor variable is set to "#FFFFFF" thoughtout the entire configuration file. It also means that the $SectionFilter variable is set to ["*sabre*",""] only in the first [[Fancy]] section (it is redefined in the second [[Fancy]] section).

The defined value of the variable is available to the theme that corresponds to that section of the configuration file. Thus, the first instance of the Fancy theme receives the values of $SectionFilter, $PageFilter, $PageName, $BGColor, $OutputPrefix, $PathPrefix, $DailyLink, and $MonthlyLink.

Theme Sections

Now that we know how to get MagicStats set up, we need to tell it what kind of statistics we want generated. The rest of the example configuration file is dedicated to creating two instances of the "Fancy" theme. Theme instantiations are denoted by double square brackets around their name: [[<themename>]]. The themename portion corresponds to a directory under your Resources/Themes directory.

As was mentioned before, MagicStats does not assign any meaning to the variables defined in the configuration file. It is up to the Themes to define which variables they use. Thus, if you look at the theme's configuration file (<themename>/<themename>.cfg) it will tell you exactly what variables it takes as input, give descriptions of all of its variables, and give default values for them. For example, take a look at the Dejavu configuration file.

MagicStats Command Line Options:

MagicStats several command line options, and although none are routinely used, they can be extremely valuable when debugging an installation, or just poking around. Lets review how a typical installation works:

MagicStats...

  1. ... is started with no command line options by the cron daemon.
  2. ... loads and parses the user's ~/.MagicStats/MagicStats.cfg file.
  3. ... loads themes and plugins found in the resources directories specified by the main config file.
  4. ... processes new hits from the specified input log files.
Given this process, MagicStats recognizes the following command line options:

  • -H - Print out version and help information.
  • -C <config file> - Override the default ~/.MagicStats/MagicStats.cfg to be used as the main configuration file.
  • -V - Run in verbose mode, printing out information about pages generated and hits processed.
  • -P - Print out all of the plugins that are found in step #3, then exit.
  • -T - Print out all of the themes that are found in step #3, then exit.
  • -N <number> - Only processes the first "number" hits from the log files.
  • -G - Print out debug information. This can be useful when submitting a bug report.
  • -lint <themename> - Run "lint" on a theme. This is only used by theme developers.

Conclusion:

This document explains the basics of how MagicStats works. Hopefully you understand enough about MagicStats to set up and maintain a fairly customized configuration that isn't straightforward variant of an "out of the box" configuration. I would really, really, really like feedback on what you think is terrible about this document, what you think is good, what needs to be explained more and expanded upon, and if I'm a total fool, please tell me! :)

-Chris

Copyright © 2000 Chris Lattner