When you have a small sample size, very small differences will be detected as significant. so you want a large sample to be very sure that the difference is real (i.e., it didn’t happen by fluke). So if you have 29 people responding the difference between 1 person saying CaseLogistix and 2 people saying CaseLogistix is so small as to be meaningless when the total number of possible respondents is 1 million (the number of attorneys in the US) But put in percentage terms for purposes of that “survey” the difference is enormous …. the CaseLogistix percentage would increase by 100% from .034% to .068%. If 5 people said CaseLogistix, their percentage becomes 17% but the increase in real numbers is still insignificant in terms of the total number of the target population.
Now if the population is changed by concentrating on say the AmLaw 100 and we know that each answer is from a different representative of that group then we can say we have a 29% answer rate from our target market and that number becomes more compelling. And if that number (5 out of 29) still seems to be too low a number to be representative of our target population, we need to get more respondents. Because that is really the question we are asking here…what sample set is significant? Using a calculator you can find at Creative Research Systems , it seems that 26 is the number you need for a population of 100 so then the 5 out of 29 becomes even more persuasive .
OK but I can see all the the technical sophisticates in the audience waving their hands, jumping up and yelling “oo oo” like Horshach on Welcome Back Kotter. (Yes Craig, I’m looking at you) OK, technically in statistics “significant” means probably true or not due to chance. A research finding may be statistically significant ( or likely to be true) without being important. When statisticians say a result is “highly significant” they mean it is very probably true. They do not (necessarily) mean it is highly important. Or as former Harvard President Lawrence Lowell once wrote, statistics, “like veal pies, are good if you know the person that made them, and are sure of the ingredients”.
Which brings us back to George and Tom, who we all know and trust. As George also said in his post, “… not everyone may realize the degree to which we must depend on self-reported information. … Much of the data provided to us, however, is not information that we can verify or refute. We have to depend on the integrity of the people and the organizations providing us with the information. At times, we feel it is necessary to give providers haircuts; we never, however, give them toupees.“
As I said above, hmmmmmmmmm. I trust George and Tom but I’m not sure about the guys with the toupees. And the guys who put on their Mardi Gras costumes before they answered the questions? You with the sneakers, out of the pool.
(for more on my own surveys of end users, see the next post. One of them is very significant statistically and one not so much, but hey, you can trust me. No really, the ponytail isn’t a weave)