What happens when the people come?
Published on September 24, 2001
Every experienced web designer knows that web statistics are important. Okay, now quick, think of a time when you responded directly to changes in web traffic in a tangible, tactical way. I'm guessing that some of you out there may be able to provide a few examples. Maybe not, though. The most common use of web statistics has always been to justify the existence of a site: "see boss, people are using the site, we can't stop now..." or to justify spending more money on a site: "see boss, we're selling lots of product. I bet we can sell more if..."
What I recently discovered is that good traffic analysis can support good web design, not just in strategic or "big picture" ways, but also in a very tactical day-to-day way. This article is going to illustrate a specific example of how some simple traffic analysis brought to light a new usability problem on one of my sites and how I addressed that problem in near real-time.
First, you should know a little something about the site in question. It's called "The Play's the Thing" and it contains the full text of the complete plays of Shakespeare. The site has a few unique features like word counts and line counts. It provides some audio of actors playing scenes and it has a search engine for spoken lines in the plays. It is intended as an easy-to-use educational resource. The design is very straightforward, employing few graphics and using standard navigation techniques to get users to the information they're looking for.
This site is listed in some web directories (dmoz.org and its sister sites). It returns well in search engines, particularly Google. During the school year I usually get between 30 and 80 visitors a day viewing between 150 and 400 total pages per day. According to my analysis of the site statistics, the usage falls primarily into a couple of patterns:
- a user comes to the front page of the site and then goes to a play and pages through a few scenes of that play.
- a user comes to the site and immediately uses the search feature -- usually more than once -- and then views a few scenes from scattered plays.
And these patterns held true until Saturday, 16 June 2001.
The pattern changes
Like any good webmaster, I check the site's statistics periodically just to see what's going on and to make sure nothing is broken. When I checked on Sunday 17 June, here's what I found for the previous week:
Clearly there was an anomaly here. Given that I had ten times the unique visitors on Sunday than I had in any of the other day that week, I decided to do some investigating.
My first thought was that the data was simply wrong. Since my statistics program tracks URLs visited, I figured I'd look at that report and find someone else's URLs listed in my report. That would have explained it.... but no such luck as you can see in the following graph:
So I knew that 500 plus users came to the site on Saturday and they used my search engine almost 1300 times. (My little 500 Mhz Pentium III Linux box that runs the database and Perl search script did yeoman duty during this mini-siege -- not a complaint heard.) My next thought was "where the heck did they all come from," so I checked the referrer statistics. I discovered that most of the users were coming from google.com so I checked the search string statistics. Here's what I found:
Now this was particularly telling. The vast majority of reported search strings were variations on "The play's the thing". (Note that the search string is not universally reported properly so the number of each phrase is less important than the percentage.) This line from Hamlet (the very end of Act II, Scene II) is quite famous. It also happens to be the title for my site. It appears in my meta-data and prominently within the content of my home page. My site returns second on Google when searching for "The play's the thing". I was getting somewhere now; I knew that a large population of users somewhere on the net was looking for the origin of this famous quote.
But why aren't they finding the answer?
I also knew there was a problem. Looking back at the "Most Requested Pages" graph. 1293 users searched (I assumed most of them searched for "the play's the thing") but only 87 ended up viewing the correct scene. This is not a good percentage. I quickly began investigating this. I found that searching for the correct full quote did not return any result (uh oh). I immediately went to the scene and discovered that there was a space between "play" and "'s" in the quote. I went back to my XML source and found the error there too. I quickly corrected the error and re-ran the Perl script that generates the HTML pages and populates the database for the search engine. I then tested the search again and found it returned the proper scene.
Now my natural curiosity was fully engaged. I had to find out why the heck my traffic spiked. I knew the technical reason, but I wanted to know the "people" reason. Why were so many people looking for that phrase? The first thing I did was rewrite my search script so it tracked all search requests. I wanted to be sure what people were looking for. Almost immediately, I saw searches being recorded for the same variations on "the play's the thing."
So I figured people were looking for that quote. I wanted to know why. I figured I might as well just ask. So I changed the top of my home page to what you see below:
It works. It really, really, works!
Guess what? No more than 10 minutes after this change went up, I got a note from someone telling me that a newspaper in Canada was running a contest and that contest required properly identifying the source for that quote. Okay then. I made one more change to the home page as follows
Want to make any guesses about how traffic patterns changed? Well I'll tell you. Even though total usage remained high on Monday (and the referring sources remained much the same as on Saturday), the percentage of users who went to the search engine went way down. Here take a look:
Recall in the earlier chart, the 35% of the page views were for the search engine; that number went down to 13% on Monday. And just to confirm this, I looked at exit pages: 61% of users left the site from the home page. It appears that users were getting their answer right up front.
As a writer, visual artist and programmer, I was drawn to the web early. (I have planned, designed and built sites for a living and as a hobby since 1994.) This background in art and words led me to a very straightforward design philosophy:
- The content is king and must always drive the design (the words part of me).
- The design has to be visually appealing (the art part of me).
- The content and design must be delivered with widely accessible, completely reliable technology (the programmer part of me).
I believe that when these three come together well, the resulting site should be truly usable. I build sites with these things in mind and I modify sites based on these criteria.
I believe this article shows that this philosophy holds true throughout the life of the site. In particular, this example points out that usability isn't always about design. In fact, I couldn't have solved this usability issue with a design change. The changes I made were entirely content-related and they made a real difference in usability.
So what am I suggesting? Well, I'll admit that this is a pretty stark example. (A half-blind Amish boy could have spotted the anomaly if I drew the graph boldly enough on his chalkboard.) I realize that we generally work on sites with much more traffic than this, but it still makes four things perfectly clear to me:
- If we pay attention to our web traffic, we can quickly spot changes in usage patterns.
- Those usage changes should be examined closely to be sure that our content, design and technical infrastructure still meet our users' needs.
- If analysis suggests users aren't getting what they need, we can often take simple incremental steps to better meet users' needs.
- If we adjust to those changes in usage quickly, we can significantly improve user experience.
And one more thing. This experience reminded me of something that I always want to remember. It never hurts to ask users what the heck is going on, and quite often, it helps quite a bit.
David Eames-Harlan is a complex man who can't possibly be summed up in a few simple sentences. When he's not doing marketing communications strategy for IBM server group, he can be found (usually at the same desk in Portland, Oregon) playing one of a variety of celtic wind instruments, writing or doing something else altogether. Huh. Maybe that first sentence is wrong.