We decided early on that we would not write our own statistics software from the ground up -- that we would use existing web statistics where we could, and write only the software we needed to integrate them with our site. We setup a logserver, whose sole purpose is to collect and analyze the webserver logs from each of our servers (we have around 20 different webservers, for the main libraries.ucsd.edu site, for our intranet, for branch sites, etc.). This logserver runs a locally-written CGI program that allows users to get a subset of the logs by server, date, and by all the fields of the webserver logs themselves (file, response code, size of file, client IP address, client OS/web browser). This CGI filters the logs and allows the user to analyze it using FunnelWeb or Sawmill.

However, webserver log analysis can only tell you so much, and leave out a crucial measurement of our site: which links to other sites our users select. To determine this, we use a very simple redirection script for all outbound links, and for all the links on our homepage. By analyzing the webserver log entries generated by this script, we can count the usage of external resources accessed from our site, and get a very clear picture of which elements of the home page are being used.

To analyze the usage statistics generated from the redirection script and to take a snapshot of our Sage database status, we wrote a statistics program to gather our redirection statistics both from our generated pages and from our search engine, query the Sage database, and output a report of the database status (how many records, how many in each subject and type, etc.) and which resources were used the most. This report is automatically generated monthly, and the data is combined with the data from the webserver log analysis to give a complete picture of our site's status and usage.