Progress: Variance vs. Trends

whiskers aren't just for saving a tasty morsel from your lunch

I recently re-read the amazing book Full House - The Spread of Excellence from Plato to Darwin by the paleontologist-philosopher Steven Jay Gould. The book was written in 1996 and should really be required reading for anyone in an introductory statistics class or who considers themselves or any part of their job to be analytical. Read it. It's worth your time, and can apply to so much beyond its two main areas of focus: baseball and natural evolution.

I've been thinking about the messages in the book a lot and how they relate to juniors tennis or development in any sport.

There are two basic types of progress in any system of rules (which is what any sport is). The first is progress toward something that is a higher quality--this is what we all typically think of as a trend or evolution. In tennis, this might be higher serve speed or hitting more winners and fewer errors. I'm sure I've said something like "evolution of the game" or "evolution of groundstrokes" in this blog. The second type of progress is an increase in variety over time. In tennis, this might be more winners AND more errors.

The beautiful insight that Gould explains throughout the book is that if you start somewhere close to a natural minimum (forehand winners for example), then by definition an evolution over time would lead to more of the "better" thing no matter which type of progress is happening. If someone has never hit a tennis ball before, they will mostly hit unforced errors and very, very few winners. After a dozen lessons, they might be hitting more winners. But if we don't look at the errors also, we can't say which type of progress they're making.

We can apply this concept to a group of tennis players in a developmental program or to an individual.

In a group, an average of the ratio for everyone's winners to errors would probably mislead us if it is the only metric for the group's progress. Maybe this is obvious--the tyranny of averages and all. But Gould's principle goes beyond just that an average can be skewed. This group is developmental. So they necessarily start with many errors and few winners. Some (hopefully many) hit more winners over the course of the program. Some will also hit more errors. But the group started with so few winners that the average winners to errors ratio necessarily increases over time whether or not there was overall progress toward better quality of hitting. There might have just been more variation.

For an individual, the reason for misinterpreting increased variation as progress would be different, but the principle the same. Remember that tennis players play against others--not just against physics or their own limits as in other sports like track and field. If a developing player begins hitting many errors and few winners, their winners will increase over time. So might their errors, depending on the opponent, the wind, the sun, their sleep, or any other number of factors. But their ratio of winners to errors goes up mechanically. Or it could be that their winners to errors ratio goes down, but only because their competition improved--limiting their opportunities for winners.

Gould argued, convincingly, that anyone serious about untangling the two types of progress should look at the amount of variation over time in addition to any "trend" measured by an average or maximum. Using the winners to errors ratio in tennis, we would want to look at a player's or group's max and min values and chart individual match stats in between. Progress might be measured by an increasing stat like serve speed. But it could just as likely (probably more likely) be measured by decreased variance of that stat over time. Do you want a higher max serve speed or a more consistent serve speed? The averages over time don't really tell you which you're seeing.

I been looking at this for all the metrics we track at Next Level Stats. Remember mean, median, and mode from 5th grade? Well it's box-and-whisker Wednesdays (and Thursdays) over at NLS!

If you're not looking at your variation within your program or your own performance, you aren't getting a clear picture. Think you're seeing progress? What kind?