Every once an a while, the sell through of one of my titles through Amazon Associates will dip to surprisingly low levels and make me wonder what I'm doing wrong. This often leads to actually doing something wrong, like making broad changes to the site or to my order page in order to "fix" the problem. I keep forgetting about statistical significance, probably because I got a "C" in Statistics in graduate school. I'm a great lover of stats, but in order for the numbers to have meaning, the sample size has to be large. For example, if you take a coin out of your pocket and flip it twice, there's a good chance you'll get the same result both times. If you redesign your business around this new found fact that a flipped coin always comes up on the same side, you're making a mistake.
It's easy to mistake noise in statistics for significant information. For example, the last ten days, the sales of my strongest title through Amazon Associates has been horrible, with a far below average sell-through of just over 4%. During the same ten day period last year, the sell-through was over 17%! It's tempting to pull up a copy of my site from the Internet Archive for last December and just replace my current version. Yet, if I compare Jan 1st to December 10th 2006 vs 2007, I find that the overall 2006 sell through was 9.44% and the overall 2007 sell through has been 10.25%. Whatever changes I made during the year, my Associates sell through for the title is up around by about 10%, though I haven't computed the margin of error for the sample size.
The difference between random noise and statistics is this. The ten day sample from 2006 was based on 58 click throughs to Amazon, and the ten day sample from 2007 was based on 73 click throughs. The Jan 1st to December 10th sample was based on 2,745 click throughs for 2006 and 2,858 click throughs for 2007. The larger sample is statistically significant, the smaller isn't. Just for fun, I went back and looked at 2005. The sell through for the 10 day period that year was just 2.74% (and I have no doubt I panicked) while the sell through for Jan 1st to Dec 10th 2005 was 10.34%.
So, I hope I've reminded both you and myself not to get excited about statistics based on small sample sizes, and certainly not to rush into altering your book sales platform based on such short term information. What's really sad is how often I fall into this trap. A quick Google search shows that I wrote about this subject in a post about publisher failures at online marketing just this past March. And this from a guy who is known for writing about book sales statistics!